devopsy.kinsta.cloud

Tag: security

  • Where Does Your Data Go?

    Where Does Your Data Go?

    One of the most interesting developments in security and compliance in recent years is the ability to follow a piece of data through an application from input to consumption and see each bit that touches it. For me, the reason this is so interesting is that it allows postmortems to actually determine exactly what was lost in a data breach. “Oh, that library was compromised? Okay, let’s see what inputs that library actually touches …” It’s so very cool, given the free-for-all that data has traditionally been in applications.

    I’m not a huge compliance fan for a variety of reasons, but this is one great result of mandating compliance. I don’t think that we would have this level of data tracking without compliance requirements driving it.

    So what can we do with this information? That’s the interesting part that I think we’ll see developing over time. If we know everywhere that a given piece of data is used, we have a powerful bit of development information. It is still early days for implementation–this is not easy stuff–but think about it. I could see a future where all redundant data access is eliminated. If we’re getting the same set of data from multiple sources, and it is being fed to the same internal consumers, then we can standardize the you-know-what out of it. A singular library to standardize formatting, a singular library to standardize look-ups, etc.

    For the most part, we currently validate and format on input. Like straight away as the data is brought into the system. That is a great idea for a system where we are uncertain where the data is going, and are sure that some attacks are coming through those inputs. But what if we discovered 10,000 more lines of code to maintain than is necessary? What if data tracing allowed us to eliminate that code by centralizing those functions? We would have to harden the routine doing the validation and formatting–because it will be getting dirty data–but no more so than we are doing literally everywhere today.

    If we can reduce source code by using a tool that is designed to improve security, is that not the ultimate shift left? The volume of source code that handles data entering the system is staggering, also. For enterprise systems, most of what we develop is actually data processing, no matter how convoluted it gets. The data store is the central repository and apps are simply accessing and modifying it. So shift left with regard to data is huge.

    Consider it. I know that you are all super-busy with other projects, but less source code lightens the load on everyone; take a look at data tracing functionality in products you already own (mostly available in vulnerability management and source code scanning toolsets), and see if it suits your needs. It might save everyone time and will help with some data-centric compliance requirements, too. That leaves you more time to make systems better.

  • Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    The attack surfaces that today’s businesses and public entities must manage have never been more complex and difficult to protect. The introduction of cloud and SaaS offerings over the past decade has caused seemingly insurmountable IT sprawl, with a third of successful attacks now occurring via unmanaged or virtually unknown assets. And while shadow IT is an urgent threat confronting many organizations, the migration of the software development life cycle to the cloud—and the rise of machine identities—is compounding the attack surface in profound ways. Emerging out of the Agile and lean strategies for software development in the 2010s, DevOps predates the mainstreaming of cloud computing.

    But once the value of cloud-based IT resources came to be widely appreciated, development teams began moving their developer environments to the cloud in a big way—to the point where many people today think of DevOps and cloud as inseparable. Recent Gartner research showed that much of enterprise IT spending is now directed toward cloud projects. In fact, for IT categories like application software, infrastructure software, business process services and system infrastructure, more than 51% of IT spending will shift away from traditional solutions to the public cloud by 2025. Nearly two-thirds of spending on application software alone will be directed toward the cloud by 2025.

    The explosion in cloud DevOps has brought with it a huge uptick in the number of privileges, credentials, identities and access rights under management. We’re all familiar with how digital identity and access management (IAM) works in conventional settings through combinations of usernames and passwords. In the DevOps world, where cloud containers can run as distinct microservice processes interacting with other processes (machine-to-machine communication), identities and privileges are usually governed through SSL/TLS certificates. These machine identities have emerged as a critical security priority across multiple industries, not just because they’ve proliferated exponentially, but because of the lack of effective tools and strategies available to IT security teams.

    Machine Identities—Scope of the Issue

    A recent global survey of 1,000 CIOs found that the average organization has 250,000 machine identities under management coming into 2022, with that number expected to double by 2024. The study also found that 83% of organizations suffered a machine identity-related outage in the past year, with 57% of organizations experiencing at least one data breach or security incident related to compromised machine identities over the same time period.

    Because the perimeter defense solutions and strategies that formed the basis of IT security best practices for many years are of little use in cloud environments, security admins have had to rely on existing IAM and privileged access management (PAM) tools to secure data, accounts, and development pipelines. Not surprisingly, identities and access privileges have become the leading attack vector in the cloud, providing attackers with an exponentially higher number of vulnerabilities to target within the enterprise than were available even a few years ago. We’ve seen these points of weakness come into play in recent high-profile breaches and attacks such as SolarWinds, Codecov and PHP.

    Embracing Zero-Trust Principles to Secure the DevOps Pipeline

    Fortunately for today’s DevOps teams, cloud-oriented privilege management technologies have evolved rapidly in recent years, especially in the categories of cloud infrastructure entitlement management (CIEM) and time-based role access management or just-in-time privileging. CIEM solutions target the security weak points and process vulnerabilities that are unique to the cloud environments. These include the lack of deep visibility into user, group, and role privileges, which together make it difficult to conduct oversight and control over the activities of users within cloud infrastructure and applications. Where conventional approaches such as IAM and PAM are designed primarily to secure data, apps, and users within the organization, CIEM is designed to manage permissions risks to prevent cloud breaches and data theft.

    Emerging CIEM solutions are bringing to bear advanced capabilities in discovery, monitoring and auditing to significantly broaden visibility into DevOps processes for IT security professionals. With a CIEM solution, security teams can scan cloud applications for existing users, groups, roles and granted privileges down to the resource level. Each user, group, role, and privilege can be categorized to identify and verify that privileged users have the appropriate levels of access. Access reports can be generated to concisely outline users and groups with the most privilege and the highest risk.

    On the audit front, CIEM solutions provide the capability to track and report on levels of privilege as well as all activity associated with those privileges. In this way, it becomes possible to generate privileged access reports for consumption by internal and external auditors to ensure compliance with regulatory and other requirements such as SOX, GDPR, HIPAA, GLBA, PCI-DSS, ISO 27001, SOC 2, FedRAMP and more. Similarly, with the ability to monitor privileged access across all cloud applications, security teams can apply behavior analysis to identify privilege abuse and abnormal activities from users. As with conventional SIEM tools, CIEM solutions will generate alerts to administrators to suspect activities or behaviors—for instance, if a user attempts to circumvent policies preventing direct privileged access to cloud services.

    Just-In-Time Privileging and Zero Standing Privileges

    Importantly, CIEM solutions are also effectively eliminating standing privileges to cloud resources based on least-privilege access and zero-trust principles. The automated granting and expiring of permissions—just-in-time (JIT) privilege grants—is highly effective at minimizing attack surfaces. Because these JIT/zero standing privilege (ZSP) solutions work on a zero-trust model, no one and nothing is trusted with standing access to cloud accounts and data.

    With JIT permissioning, elevated privileges can extend either for the duration of a session or task, or for a set amount of time. Once the task is complete, those elevated privileges are automatically revoked–all without sys-admin involvement. Where a user previously had standing access privileges potentially extending around the clock for months at a time, converting to JIT granting would compress that attack surface to several hours per month. Further, JIT permissioning largely frees organizations from having to maintain and pay for both privileged and non-privileged accounts. Dynamic secrets generation – where a dynamic secret is generated on demand and is unique to a client, instead of a static secret being defined and shared ahead of time – also provides a better model for securing temporarily deployed services and features.

    DevOps and DevSecOps are still new and fast-evolving concepts within the wider computer science and cybersecurity universe. No doubt, DevOps has been wildly successful in accelerating automation and speeding time to market for innovative applications and business services. To date, however, security solutions providers have struggled with cloud access management. In other words, they have struggled to accelerate privileged access solutions that could secure the devices, data, and resources used by DevOps teams, especially in cross-cloud environments. Dynamic permissioning platforms using JIT privilege grants and employing ZSP principles show great promise in solving these problems.

  • What Developers Need for Software Security Success

    What Developers Need for Software Security Success

    Given today’s evolving threat landscape, organizations and businesses in every sector now have a critical need to produce secure software. Criminal gangs, professional attackers and hostile nation-states are employing advanced tactics designed to exploit any vulnerabilities in programs, applications, networks and even raw code. Attackers are constantly finding new ways to bypass even the most advanced protections and defenses. For example, many have shifted their focus from delivering malware to instead compromising credentials, or launching targeted attacks against a supply chain. And while those high-level incursions are happening with much greater frequency, so too are the more basic attacks like ransomware and SQL injections that have been a scourge on cybersecurity defenses for many years. It’s becoming apparent that while cybersecurity platforms and defenses are critical components in defense against modern attacks, what is truly needed is secure code that can be deployed free from vulnerabilities. And that requires security-aware developers with verified security skills.

    Most developers say they are willing to champion security and commit to higher standards of code quality and secure output, but they can’t do it without a lot of support, as well as a reworking of the traditional metrics by which they are often judged by their employers and organizations.

    Why Most Developers Don’t Prioritize Security

    Coding best practices have continued to evolve over the years in response to business needs and market trends. In the past, most applications were created using the so-called waterfall development model where software engineers worked to get their code ready to meet an ongoing series of milestones or goals before moving on to the next phase of development. Waterfall tended to support the development of programs that, having met all of the previous milestones along the way, were free from bugs or operational flaws by the time they were ready for the production environment. But it was slow going, with sometimes 18 months or more between starting a project and getting to the finish line.

    The Agile method tended to replace Waterfall, putting a much greater emphasis on speed. And this was followed by DevOps, which is built for even more speed by combining development and operations together to ensure that programs are ready for production almost as soon as they clear the final development tweaks.

    Putting speed over security—and nearly everything else beyond functionality—was a necessity as the business environment evolved. In a cloud-based world where everyone is online all the time and mobile transactions by the millions can happen every few seconds, getting software deployed and into the continuous integration and continuous delivery (CI/CD) pipeline as quickly as possible is critical for business success.

    It’s not that organizations didn’t care about security. It’s just that in the competitive business environment that exists in most industries, speed was seen as more important. And developers who could match that speed thrived to the point where it became the primary means by which their job performance was judged.

    Now that advanced attacks are ramping up so dramatically, deploying vulnerable code is becoming a liability. The preference is once again shifting, with security increasingly becoming the primary focus of software development, with speed a close second. Bolting on security after the fact is not only dangerous, it also slows the process of deploying software. That has led to the rise of programs like DevSecOps that attempt to merge speed and security together to help generate secure code. But developers trained for pure speed can’t become security experts without a lot of help and support from their organizations.

    What Developers Need

    The good news is that most developers want to see a shift to secure coding and a reprioritizing of security as part of the development process. In a comprehensive survey conducted by Evans Data of over 1,200 professional developers actively working around the world earlier this year, the overwhelming majority said they were supportive of the concept of creating secure code. Most also expected it to become a priority in their organizations. However, only 8% of the respondents said that writing secure code was easy to accomplish. That leaves a lot of room for improvement within most organizations’ development teams between what is needed, and what is required in order to get there.

    Simply mandating secure code won’t get the job done. Development teams need training, support and a change in how software engineers are valued and judged within their organizations.

    The biggest thing that is needed is more and better training for them. And it should be customized so that less experienced developers can begin their training by learning how to recognize the kinds of common vulnerabilities that often creep into code, with lots of hands-on learning and examples. Meanwhile, more advanced developers who demonstrate their security skills can instead be tasked with things like advanced threat modeling concepts.

    Teamwork should also be emphasized so that the developer community can help one another grow their skills. Skilled and willing developers who know security should be appointed as security champions. Their responsibility as a champion will be to help fellow developers improve their skills. And while a security champion is almost always an informal title, they should be given the respect, rewards, and compensation that such an important position deserves.

    In addition to funding and supporting training programs and security champions, including giving developers enough time away from coding in order to properly participate in those programs, organizations also need to change the way that developers are evaluated. The primary metric for rewarding developers needs to shift away from raw speed. Instead, evaluations could reward those developers who can create secure code that is free from vulnerabilities or exploits. Yes, speed can be an evaluated factor as well, but first and foremost, code needs to be secure and free from vulnerabilities.

    Producing insecure or vulnerable code is no longer an acceptable risk for most businesses. And bolting on security after the fact is becoming increasingly ineffective. Thankfully, the best weapon to fight this disturbing trend is having the developer community produce secure code that attackers can’t exploit. Most developers are willing to step up to that challenge. They just need the proper training and support to help make it happen.

  • The Real Pipeline

    The Real Pipeline

    I’ve made no secret of the fact that DevOps was a game-changing advance in how the business of IT was done. But people tend to get religious about the methodology and forget the point. That’s okay, DevOps has its roots in that mentality. After years of thinking bad things about the authors of The Phoenix Project for portraying anything that slows the process–you know, like code quality or security–as bad, I have come to accept that absolutes were required to drive forward motion in the market. “We’ve always done it this way!” was king in IT, but the rest of the business was fed up. That book got the ball rolling and we’ve slowly but surely pushed back against the mentality that adding milliseconds (or even a week) to build times in the average enterprise has any impact worth mentioning. Oh, some people still claim it does, but we can convincingly make the code quality/security argument.

    It is funny to me that the first few years of DevOps were focused on “We’re gonna change your religion!”—wanting to force upheaval of the entire IT org chart without a thought to the fact that you still needed specialists in storage, ops, etc. to keep the lights on. The mentality was that development issues were a people problem. To a large extent, they were right; they just missed one people problem in the set of equations. Let’s look at the real pipeline that we all don’t talk about, but really should:

    Notice there aren’t tools in this chain; instead, there are groups of people. It is sad but true that the last two–representing acceptance testing and customer rollout–are almost always merged these days in all but a few organizations.

    This isn’t controversial; we still do the dev bit largely before the ops bit; the business has to decide what we want to offer the customer and customers are at the end of the line. And that is the key. Customers are at the end of the line. Now, I get to make you uncomfortable. Nothing in the above diagram matters except the first and last arrows. Think of it like this:

    What’s important in what we do comes down to two questions: “What does the business unit want to offer customers?” and “What do the customers expect of the business unit?” In this sense, “business unit” might be IT and “customers” might be co-workers. That changes nothing. What the business unit offers and what customers want is all that matters. The rest is implementation detail.

    Don’t believe me? It is an easy case to prove. Let’s try this one, with or without IT involvement, depending upon the problem being solved and the organization:

    The business unit determines what they want to offer, customers let their demands be known (one way or another) and the business offers the solution they think fits for the customers. As I said above, the rest is implementation details.

    What is your point?

    This leads me to the inevitable observation. Testing and security are:

    • Completely expected by users. They don’t want buggy software, and they don’t want to enter information assuming it will get nicked by hackers. There is no compromise here; users don’t “kind of want a product that works” nor do they want a product that “might be secure, but we’re not sure.”
    • Desired by the business. They have constraints on what is reasonable–the old meme of a server with the network cable cut is only funny because it shows extremes. But they absolutely do want the product or service they are offering customers to be stable and secure. “We can just spin out a new version” only works so often and for so long.

    So, your toolchains need to include stability and security tools—I would argue at every step of the process, but where and how much is really a business call. My security peeps will get upset that I called security a business call, but it is. You can choose to be non-compliant and insecure; the impact of those decisions on your organization is all on you. OTOH, of all the things that the organization (both the business generally and IT specifically) are focused on, you can afford to drop a few and get security and stability right the first time.

    Meanwhile, you’re keeping things running while we debate things like this. While I repeatedly refer to a large portion of my target audience as “implementation detail,” I’m sure you all know how much I (and many others) appreciate the rockstar job you’re doing keeping the IT org supporting the greater org.

  • Compliance Synergy – DevOps Unbound EP 25

    Compliance Synergy – DevOps Unbound EP 25

    Achieving compliance might be challenging when embracing DevOps due to the rapid pace of development. However, the agility and automation associated with DevOps might actually simplify compliance. When integrating compliance into the SDLC, DevOps teams may have to use different tools or slightly modify part of their processes to adapt to specific requirements. In this episode,Sonya Lowrence (Tricentis), Anna Murray (Rocket Software), Dan Kirsch (Techstrong Research) and Tim Johnson (Cloudbees) join hosts Alan Shimel and Mitch Ashley to discuss how to effectively integrate compliance as part of your DevOps process and the challenges that come with it. Is it possible to deliver high-quality software at the speed of DevOps whilst ensuring security compliance? The video is below followed by a transcript of the conversation.

    Alan Shimel: Hey everyone. Welcome to another episode of DevOps Unbound. DevOps Unbound is a biweekly video series that explores all aspects of DevOps. We’ve been doing this for over a year now and it amazes me the breadth and depth of DevOps topics that we’ve had a chance to dive into here on DevOps Unbound.

    The show is sponsored by our good friends at Tricentis who have been our partners in bringing DevOps Unbound to you now as I said for more than a year and have really, in every sense of the word, partnered with us to really explore, go where no man or woman has gone before and DevOps and kind of really looked at things and take them apart. So many thanks to them for sponsoring. Welcome to our show.

    We have to sort of formats for DevOps Unbound. The every other show, every other week show is about 40 or 45 minutes of a prerecorded conversation with a panel of experts. And then around once a month we do do a live roundtable of DevOps Unbound in which the stars of the show are you, our audience. And you are invited to come on, participate in the conversation and ask questions, comment, chat with each other and it’s just – when it goes right, it’s a rip-roaring conversation of a thousand different directions. But it’s a fantastic experience. I invite you to, I think the next one might be around March 15 or something. But check it out, we’ll get the dates for you. But I do invite you to come to our next live roundtable as well.

    Anyway, let’s get on with today’s show, though. Today’s show is about DevOps and Compliance Synergy, Reality or Fiction? I have some thoughts on this but before we do, let me introduce you to today’s All-Star panel.

    On my screen, he’s on the top left. I don’t know where he is on your screen. It’s my friend Tim Johnson from CloudBees. Tim, why don’t you introduce yourself.

    Tim Johnson: Thanks, Alan and thanks for having me here. As he said, Tim Johnson. I’m in product marketing at CloudBees. I’ve been in DevOps since about 2016 and a whole lot of years in security prior to that. So happy to be here.

    Alan Shimel: Thank you, Tim. Thanks for being here. Next up I want to introduce you to Anna Murray. Anna, if you wouldn’t mind introducing yourself to the audience.

    Anna Murray: Thank you, Alan. Thanks for having me here. My name is Anna Murray and I work for Rocket Software, a recent acquisition through ASG Technologies, if you see me before. I’ve spent my career in automation. So at this point, I’m a product manager for automation technologies with Rocket Software and I’ve taken teams from Waterfall project management through Agile transformations and Agile has grown to DevOps and I am passionate about automation when it comes to all of that.

    Alan Shimel: Fantastic, Anna, thanks for joining us. Next up I want to introduce you to one of my colleagues, Dan Kirsch. Dan, if you wouldn’t mind introducing yourself and telling people a little bit about you.

    David Kirsch: Thanks, Alan. I’m managing director and one of the principal analysts and cofounder of Techstrong Research. And as a recovering attorney, compliance is certainly an issue that is near and dear to me.

    Alan Shimel: Absolutely. Our next panel member is, we’re very happy to have her, Sonya Lowrence. Sonya, welcome and maybe if you could introduce yourself to the audience.

    Sonya Lowrence: Sure, thank you, Alan. Hi, I’m Sonya Lowrence. I’m the head of compliance at Tricentis and I have been in this role and working closely with DevOps actually in a couple rules now since 2018. And before that, I was a programming manager heavily embedded with the integrations and Cloud operations for about 15 years. So I’m also very passionate about how to bring this to the dev team and streamline DevOps. So looking forward to it.

    Alan Shimel: Excellent. Last but not least, my cohost for DevOps Unbound and business partner, associate for 20 years, Mitch Ashley. Mitchell, if you wouldn’t mind saying a little bit.

    Mitch Ashley: Yeah, Alan. What a great panel. I’m excited to hear everybody chat about this topic. I’m CTO with the Techstrong Group and also work with Dan as part of Techstrong Research. So looking forward to us diving in.

    Alan Shimel: Absolutely. And thank you, Mitchell. All right. So guys, this episode is DevOps and Compliance Synergy, Reality or Fiction? Let me start it off with this.

    My background is in cybersecurity, 20+ years, Mitchell and I have worked together in it. Tim, you’ve been in cyber a long time, too. I will be really honest with you. One of my most hated subjects or aspects of security is compliance. I think compliance has done as much to hurt security as it’s done as much to help security. It’s introduced this least common denominator, lowest possible bar for security where instead of doing what we should be doing, we are doing what the minimal needs are, checking it off, check that box and we move on. And then we act surprised when stuff hits the fan.

    Now, I started devopsy.kinsta.cloud because I thought DevOps was a godsend for security. Finally, a chance to get it right. Not to be bolted on, to be built in. And where security goes, compliance follows, right? Keeping that low bar here. And the promise of DevOps, especially with compliance but even more so than security, was my God, we could automate this. If we could just automate the compliance, we don’t have to worry about it anymore and we can focus on really making secure software. Because compliance becomes sort of a byproduct of security automation in DevOps.

    Now, as we sit here today, we know that that hasn’t necessarily been the outcome. Let’s leave it at that. Maybe it has been to some of you. I’m interested to hear. Don’t let me bully you. Has DevOps been a good friend to compliance or are we barking up the wrong tree? Are we chasing our tail? I don’t know how many dog analogies I can come up with here but what is the deal with compliance and DevOps? Who wants to start us off?

    Tim Johnson: I’ll jump in, Alan. You crack me up. We’ve been hearing that for forever and the dichotomies that we have to deal with, can you be compliant and fast? Can you be projectable and flexible? It’s like yeah, come on. Knock it off. It isn’t like that.

    The way we’re seeing it is DevOps is kind of going through this evolution of can we just automate the pipeline or can we get the CI part to work? And then can we automate the testing part? So it’s kind of evolutionary. And security has kind of come along, I think about 2017, John Wallace, we did a webinar with him where he’s talking about dev sec ops and saying that the sec should be silent because if you’re doing it right, the security is there. And now we’re coming to the next evolution and the next source of friction, if you will, is compliance. Because okay, we’re going faster, we’re going secure. Somebody says can you prove that to the regulators? And the way we see it is we have a bank customer of ours has 100 people devoted full-time on a 90-day rotating basis to doing compliance work.

    Alan Shimel: Sounds automated to me.

    Tim Johnson: Companies are doing. So that’s kind of a state of where we are. So from where I am, if you can’t do an audit easily, you ain’t running DevOps. You ain’t doing DevOps right in the first place. So if you are compliant and you can prove it to the regulators, you’re doing DevOps really well. That’s my opening salvo. Back to you.

    Alan Shimel: All right. I guess someone else wanted to say something.

    Sonya Lowrence: Yeah. I’m very passionate about this definitely at Tricentis but honestly just anywhere. I feel like it’s a two-part answer right now. I think I’m finding more compliance people in my space who understand, again, I come from a heavily background so for me, it’s very native to say compliance is an afterthought. If you look at it the right way, even though it may be a low bar, it’s an input. It’s a set of basic requirements. If you can accommodate that upfront, there are very few surprises. 

    And I would even extend on that and say in the world that we are moving to in DevOps and better automated tools where your CICD pipeline takes all of this into account, both your code and your Cloud environment, I really feel like it’s going to start to deliver inherently secure code and infrastructure. It’s just that that vision and the tools that are bit lagging, right? I don’t see that there are really any surprises. Couple of example frameworks SOC 2 type 2, ISA 2701. The objectives are clear. If you wrap that in at the beginning as a requirement, then there are very few people that one, don’t get trained as they go and two, can’t deliver because it actually is not that hard or complex to deliver technically.

    So I think that part of what we are seeing, and this is in part to me having been a program manager for years, Agile took and pared it down to a very lean perspective of we’re looking at features first, customer needs and typically things like encryption and administration and users was kind of an afterthought and it’s not built-in to the traditional Agile tools. I think that along with what Tricentis is delivering for quality, many companies are starting to respond with respect to compliance and security and putting it back at the front where I think it belongs. Again, this is not technically difficult. It’s the fact that it’s basically not built in to the current CICD tools. It makes it challenging, I think, for developing teams and infrastructure teams.

    David Kirsch: Yeah, I think what Sonya said made a lot of sense. It’s also, now when we are talking about DevOps, we talk a lot about culture change and changes in process as much as technology. So if the dev team is just go, go, go, go, what new features have you added today? They are never going to think about compliance. But if the development team feels that they have a working relationship with the compliance team and they are working together and they understand what the compliance team needs and why they need it, not just oh, it’s a checkbox but it’s important and it’s important for more than just our auditor to look at. That sort of merging together of collaboration between DevOps and compliance is an important aspect.

    Sonya Lowrence: Absolutely.

    Tim Johnson: I think part of that is some of the evolution that we are seeing and thinking, at least from the dev side is a maturing from a bug is a bug but it’s matured into a security vulnerability is a bug and noncompliance is now starting to be thought of as a bug. And there really aren’t too many developers out there who aren’t passionate about writing bugless code. But the challenge is, what is compliant? And how many standards do you have to adhere to? And how does that translate into all the different environments, the runtime environments, the binaries, the code, the access to the data that you’re dealing with. How do you translate all of those mumbo-jumbo from those regulatory standards down into being compliant with this piece of code that your writing? That’s a big challenge.

    Sonya Lowrence: No, I agree. And especially to that point, when I am evaluating alongside our dev teams and our product teams tools that help augment and fill these gaps, I’m definitely very aware and making sure that we’re starting from the rate baselines. There are baselines that if you work from it is kind of that magic decoder ring so all the different frameworks you need to apply going forward from a compliance standpoint. And once that’s in place, again, it just streamlines all of this for all of the teams. So it’s a really important consideration in tool selection.

    Anna Murray: And I’m going to finally jump into the fray here. I think that’s also been part of the challenge that we’ve seen among people we are talking to is that there are so many tools in place. How do you get that report so you can see what’s happened and you can reply to your auditor? And so we’ve really been talking to people a lot more about making sure that you have a plan that surrounds all of this because not only do you have a lot of tools, you might be replacing one or two of them every year because there’s better technology. And so you need good, planned workflows that are going to work for you no matter what you put in play into that workflow. And it’s not an easy answer to get from here to there and as somebody said, it is a journey for every company. And so it really does behoove the leadership to make that culture change as Dan talked about. But it isn’t just about adopting DevOps. It’s about having an end to end plan or making sure that you are delivering that value on a consistent basis as things change around us all the time.

    Alan Shimel: Agreed. To me, I’m going to be just the devil’s advocate today, I’m sorry. But I almost hear us saying hey, you know what’s great about us this DevOps thing? We are going to make compliance the developer’s problem, not the security person’s problem. Right? Developers, they’re well-paid, what the heck. But they have enough on their plate. I don’t need my developer necessarily to be my tester as I think the folks at Tricentis will clearly tell you. I don’t need my developer to be my security person. I need them to be security conscious, security aware. But I don’t need them necessarily to be a compliance expert. So therefore, if we can’t develop the tools that can automate compliance as almost a byproduct of the process and instead throw it onto the shoulders, put another straw on the camel’s back of the developer, I don’t know if it’s successful. If it can be successful.

    Tim Johnson: Agreed, agreed. The shift security left thing just really bugs the bejeebers out of me on a couple of levels. 

    [Laughter]

    Tim Johnson: One, like you talked about Alan, just the burden that it puts on the developers to know more stuff. But there’s also kind of the implication if we do all the shift to all the left, all the rest of the stuff to the right of that magically takes care of itself. Environments can be changed. Credentials can be mucked about with and stuff once the process has started. So there’s two levels here. Your pipeline, defined pipeline should have that level of governance and guardrails and gates and everything else that goes along. But you still beyond that have to look at all of the other moving parts around there and attest that everybody did what they are supposed to do. Because at the end of the day, it’s folks like Sonya or maybe Sonya’s boss who have to sign the paper that said this meets our risk profile and we can go forward. That still has to be a separate thing. But you need that visibility outside of the pipeline itself to be able to see it and control that.

    Sonya Lowrence: Yeah. And to speak to that, I feel like again I’m going to come back to people aspect of this challenge. One of my biggest partners in any company that I work at, especially when you start looking at the program and platform level, future state, making this inherent in the design are architects or I know that that’s not a role that’s very frequently filled at smaller companies, the group or team of leaders in the engineering org that fill that role of architect. If I can work with the architects upfront, again, it becomes inherent in the design. I agree, Alan, I do think it’s exorbitantly way too much data or knowledge for one engineer to consume. But if you can plan your builds for containers or images and get things like that in place with the right team leads, a lot of this comes together as an affect where they are working within a very specific, still flexible. You can still achieve the future and growth with customers that you need. But at least the infrastructure around them is really secure. 

    And a quick data point that I have because I’ve been through I think two or three SOC that I’ve looked at the secure control framework, 70 percent to 75 percent of SOC 2 audits are about the infrastructure controls. It actually doesn’t directly deal with the codes that the developers are producing for the actual app. And I feel like within the industry, if that understanding were more common and the DevOps team really supported and they had the right tools for secured builds and really making sure they could see things before going to production that were insecure at that layer around the perimeter and with user access, I feel like most of these issues around certification and compliance, there are baselines, would actually be met more consistently.

    So I feel like there’s also kind of a challenge right now of a lot of people look at those product engineers as that’s the source. Those are the people with the root cause answer. They are the ones coating the solution. But the truth is, it’s your infrastructure team in the Cloud. And that’s really something that I’ve seen on premise a little bit different. But it’s still the same statement. You’ve still got networks and IP and ports and perimeters to control. So I just feel like the infrastructure piece of this is typically left out and then it becomes, unfortunately for many companies, the tech debt when they start looking. 

    And I’ve coined this phrase recently, I’m calling it, it’s not a moment of realization. It’s becoming a moment of terror because when you do get the right tools and you can see it, everyone steps back and says I had no idea this is what we had built. And it takes a lot of fortitude to walk through that backlog, that tech debt.

    Alan Shimel: Does anybody want to comment on that?

    David Kirsch: Well, yeah. I had the same feeling that you do, Alan, about dev sec ops and we are going to have these amazing – there are a lot of amazing developers, developers who know everything there is about security. Everything there is about operations. Everything there is about compliance. It’s not realistic.

    So instead, part of it is, I think, building guardrails. So I’m sure Sonya is working on that. What sort of data are you going to allow your developers to put into the Cloud, for instance. Because at the end of the day, we’re talking about data often times in terms of compliance. So what kind of data are you going to allow in the Cloud or out of a certain region? When you’re testing applications, are you going to allow the testers to see all of the data? Probably. Probably not, I don’t think you want them to see all of your production data. What sort of role-based controls do you have in place? So setting those foundational guardrails, I think, is really important to keep the whole DevOps process clean and compliant.

    Alan Shimel: Okay. Go ahead.

    Tim Johnson: I would take that a little bit further back to the first principles of the – and this gets into culture, this gets into cooperation, this gets into silos, this gets into doing it all right is starting with a description. Can you describe your software delivery process? If you want to call it a value stream mapping exercise, call it that. But you get everybody in the room to describe how this software moves through the system then agree that’s how it’s done. And then agree on how to measure that. And then the final piece of the puzzle that I see overlooked way too often is that everybody’s compensation plans or bonus structures or measurements have to be aligned on that description of what that is. Because I’ve seen companies where the three different groups, we are all in on DevOps but the engineers are doing things differently and the security folks are doing differently and the DevOps teams are doing differently because they are measured differently. They agree in principle to doing DevOps but because they are compensated this way and they are measured over here by something else and something else, they are never going to be as successful as they are. So if you want to make this whole process as friction-free as possible, get everybody to agree and one of the things we’ve always seen in these value stream mapping exercises is somebody says and this happens and then somebody else immediately jumps in and says actually, no, that’s not how it works. And you have to get those out of the way because that’s going to come back to compliance. Well, it doesn’t work that way. Well, then how can we be compliant and so forth. Anna, I think you wanted to jump in on that.

    Anna Murray: There’s so many things to talk about but when you guys talked, first Dan and then you, it comes back to culture. It comes back to leadership. And I think it reminds me of when we were making our transition from Waterfall to Agile. It’s this go faster, go faster, go faster. And there’s more emphasis on go more faster. Go faster, go faster. Wait, slow down. We actually go slower if we don’t do the right planning. And so there’s absolute proof that if you plan well, you will actually go faster. And part of that planning must include your security and your compliance. The plan is, you can go back to your Agile definition of done. This is done when. And part of the when is your security rules. It will pass these tests. It will go through these scanning tools. Whatever your definition of done is, that’s how you make DevOps help you with compliance is you first start with the right plan. And that only happens if you have cultural leadership. And like Tim said, you get everybody in the room and you agree.

    And so it is a big challenge and I think more than anything it comes back to leadership because the people who built software love building software. They love writing code. They love delivering bug free software. That’s how they get their value for themselves of what I do every day in my life. If I deliver code that gets hacked, that puts my company on the news, I’m not very happy with myself.

    So when you think about it, leadership can put things in place and help teams go where they need to go while leaving the guard rails in. But if the leaders are not on board, you end up with this fracturing that Tim is describing.

    Alan Shimel: So to me what we are describing here, though, is quality. No one wants to produce shoddy work whether it’s secure, insecure compliance, really, we’ve really strove in security to make security synonymous with quality. If you are delivering a quality product, and whether that’s software or a guitar or whatever it is you’re selling, you want to deliver quality.

    For many – and I’m going to put forth this. For many developers and even testers, quite frankly, and folks, other folks in the software development lifecycle, security and certainly compliance are not quite synonymous with quality. Security is other to them and compliance is other’s little brother. But it’s not quality. It’s not quality. And I think that again, that’s one of the promises of DevOps is making that quality.

    I’ll throw just something else out here because it’s at the heart of DevOps and kind of this whole way of looking at things. Infrastructure is code. You know Sonya, you made an excellent point that if you set your infrastructure up right, compliance almost does become automated and easy because so much of your compliance is based on that infrastructure.

    But in the world where infrastructure is code, is code, not as code, is code, that can be changed, that can be duplicated, that can be cloned. When I first started devopsy.kinsta.cloud, puppet and chef ruled, Britannia ruled the waves because you can create that same instance over and over, that same configuration, that gold config that was rigid, it was secure, gave me compliance. That was one of the big benefits here.

    Sonya Lowrence: Absolutely. I feel like that’s why it ties back in now, like at this point in the conversation, naturally to a tenant of a privacy that is becoming more and more standard in many countries. That’s security by design is the expectation. In the world of Cloud, some of that challenges, you’ve got a lot of options and not everyone understands on that front how to build it securely. There is still a lot of base designs out there that don’t necessarily have the configuration controls that maybe they could or should. That’s debatable. But I feel like that’s the challenge here is there’s a huge explosion of growth in Cloud environments and the engineers who understand the secure tenants, architects who know how to build it consistently to those standards are lagging a bit. They are learning, too, as we go. 

    So I feel like again that’s why these tools, and going back to the understanding that if you were to look at that process like you mentioned, Tim, it’s really important to understand the SDLC. And I recently did this just for a dec. A very simple here’s how many times you have to touch code or infrastructure if you put this up at the front in a beautiful DevOps environment and approach, it’s one hand off and then your security team, ideally, is watching for vulnerabilities. But the truth is, I think most companies end up touching code or infrastructure 10 to 12 times before they see the vulnerabilities to deal with them. 

    So I feel like the more we can find the right tools to actually put this up front or working with the architects or leads to really build out that future state environment and build these automation points, we will continue to see this challenge. And it is very, very challenging for any one developer to account for all of that. These are many layers of expertise that we are touching on today.

    Alan Shimel: Absolutely.

    Tim Johnson: One of the things, Alan, I saw a statistic last week and I apologize, I can’t remember the source. But to your comment about infrastructure is code and repeatability and we can get into the concept of immutable objects and the pipeline and all that other stuff. There is a potential fallacy or vulnerability there because yes, you were starting with that. You are using this approved component and its repeatable. But the statistics said that only half of the controls from a regulatory standpoint actually covered infrastructure is code. So there were other things that were happening that your IAC solution or methodology or component wasn’t going to see. So that’s – okay, you started well. Did you continue well and did you end well is kind of the thing that you have to kind of keep along and keep in mind as you go through that.

    Alan Shimel: I don’t disagree. Mitchell, you’ve been letting people talk today. I’m interested in your thoughts.

    Mitch Ashley: Absolutely. You’ve got a great panel. It was just thinking about several things. One thing Anna said was talking about culture and leadership. And I’ve learned the hard way. Automation does not fix culture. One of the toughest things –

    Alan Shimel: You can’t automate culture?

    Mitch Ashley: No. I mean, automation doesn’t solve your culture problem. And one of the hardest things to do is to try to automate across silos. That’s something nearly impossible and you’ll end up with a mess. And so I think we kind of have to shed a little bit of the paradigms of thinking of these as fixed objects. Compliance is not fixed. It’s constantly changing more quickly. Tim and I know this from some of our research together. DevOps is speeding up things, Agile is speeding up things. We are changing our toolchain. So it’s almost like instead of thinking about we are building a highway, it’s like we are trying to do compliance for something that’s constantly in motion, that is flowing by like a river. And so we have to be really good at things like understanding the changes that are happening and understanding the data that might be helpful. Maybe it’s not the panacea that solves all of our problems. The data that we are already throwing off or can capture as part of our software development process. And understanding the tool changes and I think configurations are going to change that whether it’s infrastructure’s code or we plug in a different tool. And that’s kind of the mindset we need to adopt whether it’s compliance, security or DevOps and software.

    Alan Shimel: Yeah. Might very well be. Let me defend the common man developer, common man or woman developer. As a security person, I fancy myself a security person. I am bewildered and befuddled by the amount of compliance regulations that are out there, depending where I am, who I’m talking to and what I’m doing. How can we honestly, earnestly, expect the developer to untangle this mess?

    I’m reminded when politicians hold up those big – these are the regulations we cut this year. And they slam down this big fat book of regulations. The regulations never go away. How is a developer supposed to make heads or tails of this, though? Or maybe they shouldn’t and that’s DevOps job?

    David Kirsch: Yeah, I mean that’s really the job of different tools and platforms because you have to codify those compliance, those huge documents that what are the 1500 actual requirements? What is Regulation 43BC Subsection R? What does that actually mean? And that means you’ve got to check this little thing of code. And so that’s got to be automated into the platforms to make it easier because it’s just unrealistic to expect developers to understand it but they are not paid to care about it. They are paid to put out really high quality software. And that’s what you’re paying them for.

    Tim Johnson: A lot of money.

    David Kirsch: You’re paying people like Sonya to figure out the compliance problems and you want the developers to put out often software.

    Alan Shimel: But you know what? Anna, Tim, that sounds like your job. You guys got to be making products that do that. How do you do it?

    Tim Johnson: I’m very difficultly avoiding a commercial for the products I’m associated with right now.

    [Laughter] 

    Tim Johnson: There’s other aspects to this which you haven’t really talked about is the context of it, of whatever issue that you find. And just what the heck is the programmer or developer supposed to do about it? So there’s any number of securities scanning tools, stass, yast, blast, bast, cast, whatever. And they all come back with this storm of alerts. There’s scary things happening. Developers are what the heck do I do about it? The compliance officer, like my colleague here, Sonya, says what the heck does this mean? Because it’s just as challenging for the compliance officers and security people as it is for the developers. And you use Bob’s software that has some AI magic that says we determined this is a threat level of 563. What does that mean? And some other Joe’s software says oh, that’s a threat level of 12. And things like that. So when you discover this, what does it mean and what does the context of what it is that we are doing? So you look at things like where is this in the lifecycle? Yeah, it’s an open connection to the Internet but this environment is a test environment. It has no connection to the Internet whatsoever. So do we need to panic about this? No. If it’s in production, obviously, yes. But it should never get there in the first place.

    But also, you’ve got this issue, you’ve got to be able to feed it back to the developer in the tools that they want to work with in a way that they can do something with it. So they don’t necessarily have to understand you’re not compliant with this particular control of CIS 12, you can speak Urdu or Phoenician to them. You get better results. Here’s this thing, here’s what’s wrong. Here’s how you fix it. Go do it.

    Alan Shimel: Anna, what about you?

    Anna Murray: I echo what Tim says. It has to be clearly defined back to the developer where is there a hole? We recently were looking at something ourselves that we had found a problem and okay, wait. Does it affect this – doesn’t even matter? First you have to evaluate it. And it does matter, most of the time, but you have to have your architects in play. Your architects, as Sonya said, she doesn’t even meet with some titles called architect. If you don’t have somebody in that level, you don’t have anybody you can look at the whole product and say what the threat level is because of your product and what it’s meant for. And the access that can be to that software for it to cause a problem.

    So that’s where you really have to have those people that are at the architect level, whether that’s their title or not, and they are interfacing with your compliance people and understanding the things that need to be architected in and you have to have standards. You have to have them codified into your tools. And we are scanning and we choose to use this solution from CloudBees or that solution from Rocket. And we’re going to make sure that we meet this threat level that’s defined by that product and then your compliance people review it and decide whether it’s good enough or not. And if not, up your game. But automate it, give the reports back and you better hope those reports, as Tim said, tell the developer what’s going on. Because really, these things change all the time. And developers are good at writing code and making software work and they can make it work the way we wanted to given the requirements. But that comes back to giving them clear requirements. We cannot ask them to do more than that. We want them to be creative, building whatever it is that they are building and we give them requirements to fit. And if we’re not doing that from leadership and architect and compliance, then we are failing, not them.

    Sonya Lowrence: I completely agree. And I would like to add to that. For me, I have seen some organizations where compliance will walk into any design meeting and say here’s my list of 200 requirements and regulations or whatever their source is for their input. And I really would challenge any security or compliance professional, especially in a leadership role, to really think in terms of make it simple. Make it accessible. It is not hard for developers to understand. Data is what we are protecting. It’s kind of like you’re building a fortress around your gold as a dragon. And from there, think about in terms of what you are protecting. It’s the perimeter. It’s who has access to it. It’s principles of least privilege.

    And so I feel like approaching the development team to help them come up to speed and not terrify them with too many details is the right way to go. And it just changes the culture. It shifts naturally because they do understand these principles and they do want to deliver quality code that doesn’t have these types of issues both for security and for compliance. So I feel like there’s definitely a lot of improvement in this space for training, too. Really explaining as you go why it matters and then making sure you’re working at the right level, finding those leaders on the engineering side to partner with.

    Alan Shimel: Training around – I’m sorry, go ahead, Tim.

    Tim Johnson: Well, I was going to jump back in. On that standpoint, we’re talking about the developers, we’re talking about a number of people. But part of this equation is the leadership of the organization. And you talked about, Alan, you talked about quality. And is quality enough of an incentive for upper management to really get behind compliance efforts and security efforts. And I would argue that no, it’s not. Typically, it’s kind of nice to have. And I remember statistically, from a couple years ago where managers have said they’d rather go fast than secure. This was only like three or four years ago. That’s scary. I hope it’s changed.

    But one of the things in talking with some of our customers and prospects in the last six or seven months is the concept of a tax, the compliance tax. One customer actually used that phrase. And we go back to this bank that I mentioned earlier. It’s 100 people year-round devoted to doing that. What is your fully burdened cost of that labor to do zero value add work? Right? Compliance is something you have to have but it doesn’t add any value to the organization whatsoever until you don’t have it in a big way. Then it takes a lot of value on. But that’s kind of funny money because you’re going to pay them anyway.

    But if you also add in the concept of what should they be doing in terms of technical debt or the opportunity cost of them spending this time on zero value at work versus value add, so you take that fully burdened cost, what are the expected returns? So if your fully burdened cost is $100,000, I think the company wants you to generate more than $100,000 in value in a year. Maybe it’s $200,000. Maybe it’s a total revenue divided by total employees. Some figure that’s more than what you are paying them for. And you add that into the equation of these 100 people doing this work. That’s the labor costs. Then here’s what we are missing. Here’s the value to the organization we are missing from that. Then you get a much more comprehensive picture of the true cost to the organization and how to justify the efforts of changing culture, doing those meetings that nobody wants to go to and putting in different metrics and things like that because you have an measurable impact that you can have on the organization. It’s like a snapshot on day one and then six months later, you halve the people doing compliance. You’ve got a defendable metric to take to senior management and say this is what we’re doing.

    Alan Shimel: I mean, being able to actually show compliance is, it’s a little different than being able to actually show security where if you did good, no one knows because that means you didn’t have an incident. But with compliance, you can say hey, look. I’m compliant with SOC. I’m compliant with PCI, whatever it is, GDP or whatever you are compliant with. And you can hold that up.

    Anyway, team, panel. We are over time here. I knew this was going to be one of those sessions where we ran over. And for me, it’s a very volatile subject. Sonya, as the true compliance person, not making a product around compliance or anything like that but really where the rubber meets the road, I was going to give you the last word today on today’s show to leave with our audience in leaving with compliance and DevOps.

    Sonya Lowrence: Yeah, absolutely. I think just to summarize a lot of what I’ve heard here today and what I’m trying to really make sure we are influencing here at Tricentis, you know that true realization that along with leadership, compliance is an absolutely important part of the team. And when done well and when done from the right approach is whatever tools you’re working from. You start with what you have, you identify the challenges and you get better. A lot of this boils down to knowing your requirements, following a pretty classic Deming cycle: plan, do, check, act. And I feel like those tenants really have never changed. I just hope that more organizations really start to add this back in within Agile development environments that we all have. So I think it’s very achievable and the tools are catching up quickly to get us there.

    Alan Shimel: Excellent. Well, Tim, Dan, Anna, Sonya, Mitchell, thanks for being on our panel in this episode of DevOps Unbound. We’ll be back, as I mentioned, every other week so in another week or another two weeks we will be back with a fresh panel and a fresh subject and something DevOps related. But until then, many thanks to Tricentis for your sponsorship. Many thanks to you out there watching this. We appreciate your time that you set aside to watch this. And we are grateful for it.

    Until then, though, this is Alan Shimel for Techstrong TV, DevOps Unbound. Have a great day, everyone.

  • What a Dev Wants: Security Training for Today’s Developer Workforce

    What a Dev Wants: Security Training for Today’s Developer Workforce

    It’s no secret that developing a strong security culture is a critical component of the production of secure products, yet organizations often overlook the fact that their security training programs set the foundation on which this culture is built. When it comes to developer training, our goal should be to align the intrinsic motivations of a learner with the company’s strategy and objectives. But if either of these is misaligned, the training program, and ultimately the organization, will suffer because it is either irrelevant or perceived as a compulsory box that needs to be checked.

    Addressing the Interests of Developers Vs. the Organization

    Unfortunately, there is a gap between the buyer persona and the developers themselves, even though developers tend to have a fairly consistent response about the ways they want to be trained. Rather than implement a program that is doomed to be viewed as a time-consuming compliance requisite, organizations should rather ask this simple, yet pertinent question: What do developers really want out of their security training program?

    When it comes to today’s security training programs for developers, there’s often a separation between compliance-based training and developer-centric training; in other words, the interests of the organization don’t always align with the interests of the developers. In most highly regulated organizations, there is a mandated security training curriculum that all employees, including the development teams, are required to go through on a yearly basis. This leads to a drawn-out, painful process that leaves employees clicking through modules as fast as possible. Developers may then need to go back and look up security topics on a regular basis if they are looking to implement them into their code.

    While developers are certainly making an effort to adhere to security requirements, this greatly slows the ability to get the product to market in a secure and efficient way.

    What Training do Developers Really Want?

    Organizations should prioritize programs that will specifically target their development teams by asking what those developers are actually looking for. With so many training programs on the market, it can be difficult for an organization to determine which will resonate the most with their employees, specifically their development teams. The fact of the matter is that, when it comes to security training, the majority of developers want to do as little of it as possible. They are willing to check the boxes for compliance purposes, but they want it to be as seamless as possible. As the complexity of training tactics continues to evolve, an ideal curriculum is simple, like programs that can be embedded in the tools developers use on a daily basis.

    The most effective training programs are engaging and relevant. Managers tend to see better results from training that is hands-on, interactive, engaging and related to developer tasks. Gamification is also a great way to engage developers. For instance, organizations have started livening up their training curriculum with leaderboards. In fact, recent studies have indicated that developers would opt into shared company leaderboards with points for content consumed. Not only are these programs engaging for the developers, but they are also incredibly effective, as they motivate teams to view more content by leveraging individuals’ pure competitive drive.

    Security Champion Programs for the Win

    Organizations could also implement a security champions program. These programs vary depending on the organization, but in its simplest form, a security champions program designates a “champion” from each team. The champion is then tasked with leading the team as a subject matter expert as far as security goes. One thing to keep in mind here is that this program could be a major failure if done improperly. Remember, you don’t want to designate just anyone as the team security champion; you want to be sure you’re selecting an engaged employee that will prioritize secure practices and proactively communicate updates across the team.

    Regardless of what avenue an organization chooses, the name of the game is ensuring development teams are actively engaged with a relevant and realistic training program. The result of which will have an important downstream impact as organizations work to build a culture with a strong foundation in security. After all, “checking the box” can only go so far when it comes to developing a winning—and secure—solution.

  • The Secret to Secrets Management: Reduce Source Code Vulnerabilities

    The Secret to Secrets Management: Reduce Source Code Vulnerabilities

    We are in the cyberattack age. From Microsoft to Crypto.com, no company is completely safe from malicious attackers, and it seems as if a new major cyberattack is announced weekly, if not more frequently. As more companies not only go digital but expand their digital offerings and initiatives, there are more opportunities for cyberattackers. Security teams within IT organizations are hard at work to ensure that their company isn’t next in the headlines. However, as the tech world moves to everything-as-a-service (IaaS, PaaS, SaaS, etc.), the move toward everything-as-code is happening in tandem. As a result, many developers are unknowingly keeping sensitive data (secrets, passwords, certs, keys, etc.) stored in Git source code repositories. This information is often pushed to production which puts sensitive information at greater risk of exposure.

    The good news is there is a way to ensure that secrets are locked tight within source code. Let’s explore the complexity and realities of securing source code today and how to take the guesswork out of finding at-risk code that could land your company on the front page of the news tomorrow.

    Secrets Management is Challenging and Costly

    Sophisticated cyberattackers will stop at nothing to breach valuable data assets. Secrets leaked from source code make it easy for threat actors to access applications using the same credentials intended for legitimate purposes.

    At the pace of continuous software development, developers must move quickly to build applications. However, security teams that are expected to prevent breaches are not often well-versed in all facets of the software development life cycle (SDLC). This becomes an issue as the number of lines of code (and the secrets kept within) rise alongside the number of apps developed. According to a recent report from 1Password, “two in three (65%) IT and DevOps employees estimated their company has more than 500 secrets.” And, according to the same report, “the average cost of a secret’s leak is $1.2 million. Poor secrets management can result in organizations losing $8.5 billion annually.” Secrets are vital to building any connected application or process. If hard-coded or left unprotected in source code, the proverbial keys to the kingdom are at risk of exposure—as are your loyal customers.

    It’s clear that source code security should be a priority for security teams and DevSecOps professionals. Let’s examine how they can achieve this without sacrificing developer productivity.

    How to Reduce Threats Through Secrets Management Best Practices

    Manually searching source code repositories for vulnerable secrets can be almost impossible. However, there are some best practices to follow to ensure vulnerabilities are caught early and often:

    ● Identify and remove secrets in code: You can’t protect what you can’t see. It’s important to have visibility into source code to be able to see where vulnerabilities may exist. There are a few ways of potentially going about shining a light on vulnerabilities. One is through manual scanning of source code, but this takes away valuable time developers could be spending on more innovative tasks. There are also some open source solutions that can help with scanning source code repositories, but as with everything open source, this means developers will have to maintain it (keep it up to date, integrate it, etc.). Instead, security teams should consider looking for a unified toolchain platform to help with source code vulnerability identification and remediation.
    ● Alert and remediate in code repositories: Provide a fast feedback loop to a developer or DevOps engineer before releasing code into production. Remediation should happen at the time of commit. The idea here is to block secrets from being committed to your source code repositories in the first place.
    ● Adopt a secrets management tool: Don’t store secrets or binaries within source code—nothing but code should be in your source code repositories. Remove secrets to a secrets manager (i.e. HashiCorp Vault), keys and certificates manager (i.e. Azure Key Vault), etc. Modularize and isolate these managers, but make sure they still communicate with each other so that they don’t expose secrets.

    As cyberattacks continue to increase, it is imperative that security teams take a hard look at their source code to ensure that it is locked up tight. Keeping secrets out of source code entirely is recommended in theory but difficult to ensure in practice. Following the above best practices will help reduce the risk of exposure of sensitive data and reduce the risk of losing loyal customers and revenue dollars.

  • GitLab Research Shows Security Concerns in DevOps Teams

    GitLab Research Shows Security Concerns in DevOps Teams

    The threat of breaches is top-of-mind for many DevOps teams, and while security continues to shift left it appears to be a driving force for many decision-makers when it comes to choosing a DevOps platform or other technologies.

    These were among the results of a GitLabs survey that also found getting security right is the number one challenge for DevOps teams. 

    Getting it Right

    Among the top reasons respondents said they chose DevOps were better code quality, developer productivity and operational efficiency—cited by 37% of survey takers, followed very closely by better security/more secure applications.

    The survey also revealed that 53% of developers said they are fully responsible for security in their organizations, a 14% jump from 2021. 

    Michelle McLean, vice president at Salt Security, said it is risky for organizations to depend on development teams alone for security.

    “Security and developer teams need to collaborate and work together–at every point in the application life cycle–not just development,” she explained.

    For example, security teams must bring runtime insights back to development so they can be applied for education and training.

    “It’s fundamentally important to choose a DevOps platform that either has security capabilities built-in or that can easily integrate with security platforms to facilitate collaboration by security and DevOps teams,” she said. “Otherwise, organizations run the risk of pushing out unsecured software or introducing other risks into the software supply chain.”

    Tim Mackey, principal security strategist at the Synopsys Cybersecurity Research Center, added that an obsolete mantra in the cloud-native community held that security wasn’t a high priority as proper DevOps workflows meant that incremental changes were small and easily deployed.

    He said the Log4j and Log4Shell experiences called that philosophy into question with a recognition that security issues can, and do, occur out-of-band from a release or feature and that continuous monitoring for external threats is part of a properly functioning security process.

    “By implementing monitoring for new threats, regardless of source or nature, the ability for cloud-native development to quickly issue tested patches becomes itself a security feature,” he explained. 

    Understanding the Pipeline

    McLean pointed out that while there is a growing understanding of security in the pipeline with DevSecOps, it’s not enough.

    “This is particularly true, for example, in the case of APIs. Because APIs aren’t just straight code, you must see them being exercised to spot logic flaws,” she said. “You need the ability to monitor them in runtime in order to spot anomalies and find areas where APIs might be exposing critical data. Although developers write APIs, their testing often falls short for security.”

    To accurately see what’s going on with APIs, developers need to exercise the API–even in pre-production.

    McLean said this is an area where security and development collaboration and integration is essential–APIs underpin a significant portion of the modern economy and the disconnect between security and development teams can have a direct economic impact.

    John Bambenek, principal threat hunter at Netenrich, a security and operations analytics SaaS company, added there has always been a delay between the release of new platforms and the awareness that security capabilities for those platforms need to be implemented.

    “With cloud-native and serverless technologies, that delay is as short as I remember in my career,” he said. “I greatly prefer the world where everyone at least understands the need for security instead of having to spend time getting buy-in that it matters in the first place.”

    He said if you can manage and implement security in a seamless and efficient way early in the development process, it is far easier and cheaper to address issues than addressing them after code has already shipped—and that’s without adding in breach or liability costs.

    “You can either fix it in dev or in prod, but you’re going to have to fix it sooner or later,” he said. 

    McLean noted many organizations are still maturing in their DevOps practices, not to mention their DevSecOps initiatives.

    From her perspective, educating developers as to what is expected of them from a security perspective before a line of code is written is key.

    “To help increase compliance, we’ve begun to see organizations taking steps to introduce security tooling into their pipeline process and add testing into the build process,” she said.

    Then they’ll add a step that triggers an alert for non-compliant builds but won’t prevent the build. Then they’ll add a step that mandates sign-off by the developer that they’ve seen the security alert for a non-compliant build but still won’t interrupt the build.

    “In rare organizations with highly mature DevSecOps processes, they’ll fail the build for non-compliance,” she added.

    McLean said organizations need to understand what’s realistic given their maturity and risk tolerance and implement controls that match.

    They also need to reevaluate those controls to look for opportunities to increase adherence to security policies and other compliance regulations.

    Overall, continuing to educate through compliance automation in the software development life cycle (in development, testing and deployment phases) further helps reinforce best practices while ensuring issues are uncovered before they become a risk or technical debt.

    Mackey said because DevOps platforms touch the software powering your business, when choosing any DevOps platform, the security of the platform itself and the competencies it enables should always be must-haves.

    “In effect, any decision about new software should be based on how it improves the current security capabilities of the business,” he said. 

  • Four Secure Coding Best Practices for Mobile Apps

    Four Secure Coding Best Practices for Mobile Apps

    Mobile apps have become the primary point of innovation for many companies, and skilled mobile developers are in high demand. The business asks developers to innovate fast using the latest smartphone features and capabilities to drive high download rates, wow their users, capture customers and grow their business. Dev teams facing constant pressure to move faster may push security to the back burner, focusing on required features and delivery dates the business demands. However, many dev teams have figured out how to innovate faster with security built in by ensuring that devs understand key security requirements and coding best practices. Working with hundreds of development teams on securing thousands of mobile apps, we found four common areas of security failure that can be addressed easily.

    Use SSL via HTTPS

    Mobile app devs may instinctually use Hypertext Transfer Protocol (HTTP) for network communications. However, HTTP puts private user information out in the open for threat actors to intercept. Developers can fix this common mistake by using HTTPS instead, which encrypts data sent to and from servers via industry-standard SSL. Android developers can use the available NetworkSecurityConfig file to set up a predetermined configuration for all network connections made within the app or manually incorporate HTTPS. iOS developers can use App Transport Security (ATS) as the default feature that enforces secure communications in iOS apps and prevents any insecure connections between the mobile app and the server from being completed.

    Validate the Contents of the Certificate

    Certificates add an additional layer of security to HTTPS connections by enforcing additional validations when performing a connection. Certificates include the certificate authority (CA) that signed it and the list of hostnames known or accepted by the application. Apps that validate these components when performing connections with the server significantly reduce the risk of a man-in-the-middle (MITM) attack that can steal credentials and sensitive data.

    To verify a certificate has been issued by a valid CA, Android developers can reference the preconfigured list of CAs included on devices running the mobile operating system. Android developers can implement native classes such as HostnameVerifier to achieve proper hostname verification within their app.

    iOS developers can leverage ATS that provides built-in validation, or use methods in the NSURLSession class to manually code these instead.

    Avoid Hardcoding Resources of a Mobile App

    Attackers often use hardcoded info within the source code of a mobile app to take advantage of users. For instance, attackers can use credentials stored within app files to gain access to a user account. Attackers can also find hardcoded API keys or URLs to gather private data or take over an app entirely. Mobile app developers can prevent these areas from being compromised by not hardcoding keys, passwords and URLs into the source code. Encrypting transport and decrypting data on the backend also adds an extra layer of security.

    Use the Latest Cryptography to Protect Mobile Users

    Outdated cryptography algorithms like SHA-2, RC4 and DES allow attackers to easily break a seemingly secure mobile app. Using older algorithms may also make a mobile app non-compliant with industry regulations leaving an organization at risk of fines or legal jeopardy. Devs can avoid this problem for Android and iOS mobile apps by using the latest cryptography algorithms available, selecting those that are suitable for a specific app scenario. When signing your binary before publishing it to the store, use keys with a length of at least 2048 bits (preferably 4096 bits), and on Android use SecureRandom or SecRandomCopyBytes on iOS when generating random values for cryptographic implementations. Android devs can leverage the Keystore class and iOS devs can leverage Keychain services to store highly sensitive data. Devs should avoid insecure modes of operation, improperly generated cryptographic keys and initialization vectors (IVs) to guarantee that the information that is encrypted cannot be decrypted by a malicious actor.

    Today’s mobile app users need confidence that mobile apps are built with security in mind. While building innovative mobile apps, developers have a responsibility to learn a practical set of secure coding best practices to protect their users (and their businesses).

  • Preparing Your Android App for an Independent Security Review

    Preparing Your Android App for an Independent Security Review

    As of July 20, 2022, Android developers publishing new or updated mobile apps in Google Play must declare how their apps collect, share and secure data. The new Google Play Data safety section provides greater transparency and enables users to examine an Android mobile app’s security and privacy practices to decide which ones to download. Developers can take an extra step to safeguard user trust and demonstrate a high standard of mobile security and privacy by undergoing an optional independent security review. Founded by Google and industry security partners, the App Defense Alliance (ADA) established the mobile application security assessment (MASA) program to validate mobile apps against a globally recognized industry standard.

    Mobile app developers can work with ADA-authorized labs to have their apps assessed against the Open Web Application Security Project (OWASP) mobile application security verification standard (MASVS) L1 requirements to gain the MASA validation. Developers whose mobile apps have passed an independent security review will be able to highlight that distinction in their Google Play Data safety section with the independent security review badge.

    Mobile security experts from an ADA-authorized lab use the MASA specification crafted from the OWASP mobile security testing guide (MSTG) through a mix of automated and manual application security testing to independently assess the Android apps against a set of checks in six key areas.

    To prepare for their independent security review, developers should follow these secure coding best practices when building their Android mobile apps.

    Android Data Storage: Security and Privacy

    ● Use the Android Keystore system to securely store credentials including passwords, private keys, cryptographic keys and other information.
    ● Do not store sensitive data in external storage.
    ● Do not write sensitive data to the system or in application logs.
    ● Do not cache sensitive inputs, such as password fields.
    ● Mask sensitive data during user input.
    ● Fill out the Google Play console data safety section accurately, including both app code and SDKs.

    Encryption

    ● Check first- and third-party code to confirm the use of industry-standard cryptographic libraries and algorithms (with recommended settings).
    ● Avoid outdated cryptography algorithms (such as MD5 or SHA-1) for sensitive data or authentication routines.
    ● Do not hardcode cryptographic keys or initialization vectors (IVs).
    ● For security, use pseudorandom number generators (PRNG) compliant with FIPS 140-2 security requirements for cryptographic modules.

    Authentication and Session Management

    ● Ensure API endpoints that provide access to sensitive data require authentication.
    ● Use randomly generated session identifiers.
    ● Ensure sessions terminate at a remote endpoint when users log out.
    ● Include password complexity requirements enforced on both the client and server sides.
    ● Enable brute force protection for authentication endpoints.
    ● Ensure tokens have a reasonable expiration/timeout.

    Android Network Communications

    ● Transmit all app data over TLS.
    ● Ensure the app verifies the validity of the certificate during the exchange.
    ● Declare all transmitted data.

    Platform

    ● Do not request unnecessary app permissions.
    ● Validate any input your app accepts from external sources and sanitize if necessary.
    ● Ensure sensitive functionality is protected against unintended access for security and privacy.

    Code Quality

    ● Remove all debugging symbols from the app and libraries.
    ● Remove debugging code (such as hidden settings).
    ● Disable verbose logging.
    ● Account for third-party SDKs in the data safety section.
    ● Update third-party SDKs and remove critical vulnerabilities.
    ● Enable PIC and stack smashing protection in apps and libraries.

    Obtaining a MASA validation from an ADA-authorized lab gives developers a competitive advantage to drive more downloads by showcasing their commitment to mobile app security and privacy. When given a choice, mobile app users are more likely to download and use Android apps with the independent security review badge in Google Play. And enterprises with higher risk control requirements like finance, health care, energy and government are more likely to purchase and use these verified apps.