Tag: Devops and Security

  • Security: Are You Doing DevOps Right?

    Security: Are You Doing DevOps Right?

    Phil Kernick, co-founder and CTO of cybersecurity specialist CQR Consulting, has no fundamental problem with DevOps, but asks, from a security perspective, “How many people do it right?”

    If DevOps is going to work and produce secure systems, then developers must take responsibility for security. It’s not something that can be treated as an additional process.

    But abstract responsibility isn’t sufficient. It’s rare for developers to have security tools in their kit and an understanding of their results, said Kernick.

    Learn From History

    As former systems administrator and past president of the Systems Administrators Guild of Australia (SAGE-AU, now the Information Technology Professionals Association), Kernick suggested that DevOps pretends that the discipline of systems administration does not exist and merely treats infrastructure as if it were a software library.

    Spinning up an instance is easy, but do you know that it is properly designed and maintained? This is a real concern, given the number of people working as developers without professional training, Kernick explained.

    Repeatable loops and short development cycles do make sense. But are you confident all the individual pieces being assembled were built properly?

    That raises questions about the software supply chain. Where did that Docker container actually come from? Is it well-maintained?

    The process needs to be managed, yet, people at some organizations are “just grabbing things from anywhere,” Kernick said.

    “It almost integrates the ‘not my problem’ attitude with development,” he noted.

    Developers tend to care about whether a component works from a functional perspective, but an untrustworthy piece of code will compile and run.

    Developers working in a DevOps environment need to understand the tools that help ensure code is secure, and they need to understand infrastructure, said Kernick.

    ‘Code Reuse is Bug Reuse’

    DevOps makes the old joke that “code reuse is bug reuse” even worse, said Kernick, citing the example of a piece of vulnerable code shared on the Stack Overflow site and reused more than 100,000 times in GitHub-resident projects.

    “I don’t want DevOps running in my bank,” he said. What he does want is well-engineered, secure, well-run, certified software.

    Sacrificing security might save some time and get a new product on the market sooner, which might be acceptable if there are no real risks to the user.

    It is easy to tell if a system is really important, as it involves tradable value, safety or privacy. Obvious areas include banking and health. Similarly, it is clear that a website that allows access without requiring any credentials is unimportant.

    But there’s a wide middle ground where it’s “really hard” to determine a system’s importance because it is so difficult to imagine how criminals might exploit it.

    For example, Kernick once bought a DVD cataloging app to help avoid duplicating items already in his very extensive collection. He thought this fell into the “unimportant” category. But—fortunately, as it turned out—he took the usual precaution of using a unique password for the associated service. Sometime later, he received a sextortion email that proffered that password as supposed proof that the criminal had gained control of his computer.

    Worryingly, when he told the developer what had happened, it did not accept that there was a problem. The company said the password quoted in the malicious email wasn’t his password for the service—proving in the process that it was storing passwords in plain text!—even though he explained he changed his password as soon as he received the message. Predictably, he uninstalled the app.

    Clearly, Kernick didn’t know what development methodologies that company used, but that’s not the point. The anecdote shows that a seemingly unimportant system can quickly become important to the user in situations outside the developer’s control. Consequently, security is always important.

    DevOps and Security

    He suggests that as long as an organization has the right security and governance processes in place, it won’t matter what development methodologies are used. So it can safely adopt DevOps or whatever else is most efficient in the circumstances.

    It is possible to use DevOps practices to achieve speed without sacrificing security, Kernick said. But to do that, organizations need to take advantage of advances in software development without throwing away hard-won security experience.

    — Stephen Withers

  • Has DevOps Caused the Re-emergence of Shadow IT?

    Has DevOps Caused the Re-emergence of Shadow IT?

    Shadow IT is back, and this time, it’s in the DevOps toolstack

    “One of our clients conducted an audit and found more than 10,000 Jenkins instances across their IT landscape!” This remark from a tooling vendor that helps enterprises centralize their continuous integration infrastructure really took me by surprise. For starters, I was impressed how the company had unleashed creativity at this scale—if the number of Jenkins instances in an organization is an indicator of DevOps. Clearly, its employees have taken the saying, “Don’t ask for permission, ask for forgiveness,” very seriously.

    However, there are certain risks with this approach, especially security risks if teams haven’t considered the long-term implication of their decisions. For instance, how do you handle another major security flaw such as the OpenSSL Heartbleed bug? You may have to send hunting squads to track down all the instances of Jenkins (in this case) across your IT estate—including the Linux server hidden away under the desk of the engineer—and patch them. If you are lucky, you can audit all the virtual machines on your chosen cloud where all your tooling instances reside. And if you are really lucky, the entire DevOps tooling infrastructure would have been set up in a DevOps way, using infrastructure-as-code principles with automated patching capability.

    Consider the nightmare scenario for IT compliance, security and governance teams: The spectre of shadow IT is amplified due to poor adoption of DevOps! Shadow IT in the application stack has always been a big headache for CIOs; the use of IT systems within an organization without the approval, or even the knowledge, of corporate IT has never gone away. Shadow IT has been historically associated with business users who made the purchase, but did not have the expertise to maintain and upgrade the software to make sure there were no risks to the wider organization. Much has been written about the problem in the application stack and how DevOps ways can help resolve the issue.

    However, with the proliferation of DevOps tools in a localized way, the shadow IT problem is now manifesting itself in the tooling architecture of enterprises. With the increasing adoption of DevOps practices and tools in the enterprise, teams have started installing various DevOps tools in a localized and autonomous way. This is leading to the shadow IT problem in the tooling landscape of enterprises. It’s slightly different than the problem that originated in the application stack, but with similar risks and challenges. Access to cloud in the enterprise has made it easier for teams to set up their own localized DevOps tool sets. A sizable program of work in its full DevOps glory can easily use dozens of tools and need to consider aspects of security such as:

    1. What is the patching policy ?
    2. Are your DevOps tools critical to the business due to the dependency on them for the build and deployment of critical  services that the business requires?
    3. How resilient is the tooling infrastructure ?
    4. Is there enough knowledge in the enterprise to handle this change?

    Essentially, there are four risks of shadow IT due to DevOps tooling, based on a pertinent classification by Computer Weekly:

    1. Software asset management: The risk of discovering unapproved and unlicensed software is huge and the CIO could be potentially jailed for sanctioning usage of unlicensed software.
    2. Compliance with industry and government standards to demonstrate quality to customers.
    3. Lack of testing and change control: Managing cycle of change and releases becomes complicated when there are dependencies on third-party SaaS.
    4. Configuration Management: IT teams rely on the veracity of information in the configuration management database CMDB and shadow IT compromises it.

    Organizations typically respond to the problem of shadow IT with rationalization, migration or even sunset of the targeted applications—most of which are reactive measures. In the context of DevOps tools, an extreme way to deal with the problem is by taking control of all tools with an “iron fist” and shutting down everything—at the risk of killing off innovation in the enterprise. In the short run, teams do achieve higher levels of productivity by deploying tools in the way that suits their business context and experience of the engineers in the team. However, over time I find that the teams are overburdened by the maintenance overhead of DevOps tooling and maintaining a tooling stack in addition to their software application stack. At times, if they are early adopters in their organization, they tend to be victims of their own success: by supporting other teams with their tooling solutions, almost becoming a tooling platform provider internally, and diverting their attention away from their core purpose of delivering value to their business partners.

    One pragmatic way to resolve this is to add an element of centralization or “platformization” of the tooling infrastructure.

    Autonomous teams understandably fiercely resist any attempt to centralize services in the enterprise. A few useful ways to gain a balance between innovation and control include:

    1. Provide a responsive tooling platform: “You can have any color as long as it’s black” approach to DevOps tooling will only lead teams to finding their own unique solutions.
    2. Anticipate tooling needs and build them: Think ahead, track the trends and procure the tools that you expect teams to achieve. For example, if a few teams are already playing with docker and containers, then think a step ahead and look at Kubernetes to manage the containers.
    3. Focus on user experience: Think about the ease of subscribing to tools, installation, billing … the entire life cycle
    4. Offer help to transition to the tooling platform.
    5. Run an internal communications campaign on the tooling capabilities that are available to teams.

    Ultimately, a well-thought-out DevOps approach and implementation can actually help resolve the challenge of shadow IT that organizations face, instead of accentuating the problem.

    — Aditya Vadaganadam

  • Cloud Services Demand Security Up the DevOps Stack

    Cloud Services Demand Security Up the DevOps Stack

    In the era of soaring use of public cloud by companies, software delivery is moving away from an on-premises install approach to a cloud-based subscription model through software as a service (SaaS).

    Customers expect a lot when it comes to SaaS products and services. Whether they are investing in SaaS-first offerings or shifting workloads to cloud-based providers, companies no longer have upfront costs for hardware and related software or a reliance on costly labor to install, manage and secure it. They expect software that is easy to launch, upgrades that are unobtrusive and feature-rich and limitless scaling to meet their changing business needs.

    When it comes to security, the cloud-based subscription model significantly changes the paradigm of how software security should be treated. Fortunately for many cloud providers, customers trust that a strong foundational security and privacy posture has been built into to the service by design. Preserving that trust requires a healthy tension of shared responsibility between the security teams demanding it and the cloud providers offering it. A new paradigm shift requires a deeper acknowledgement that an emphasis of the security problem “up the stack” to the application layer is needed.

    For security teams, ensuring the protection is built in from the beginning and moves dynamically in lockstep with the workloads requires a much closer partnership with application developers and DevOps teams. This ultimately leads to a richer understanding of the business applications by security teams, creating a more cooperative business-aware and security-minded culture. Getting there will help reduce the tension and improve the overall delivery of secure cloud services regardless of where you sit in the cloud provider/consumer ecosystem and ultimately enables value creation to the business. To get there, a strong partnership must be fostered.

    There are obstacles to moving up the stack. In most companies, the reality is that many security teams are not well-aligned to keep up with this change while the business continues to make cloud-based investments, making the problem worse. Security’s comfort zone seems very much still focused on perimeter defense, but this emphasis isn’t optimally aligned to a cloud-first future. Even when forward-leaning security shops realize a need to change, they often get pushback from app developer teams because security reviews can delay deployment and security tools create “noise” and false positives for app developers. Also, traditional security tools often do not work well in cloud environments.

    Interestingly enough, the No. 1 concern of app developers is preventing data breaches and cyberattacks, a survey of 500 developers and development managers by Veracode found. Less than one-quarter of developers said meeting customer or regulatory compliance is their top concern, while 21 percent said meeting budget and delivery schedules is their top concern, and 19.4 percent said delivering secure code to pass internal audits is a priority. Furthermore, app developers who work with security teams get “major and tangible business and technical benefit,” according to a survey of 150 security and app development pros conducted by Forrester Consulting. Almost all of the collaborators said that working together leads to major or moderate benefit in customer satisfaction.

    Respondents said the more frequently they collaborate, the better the results, “including improved customer satisfaction, business/IT alignment, quality and frequency of releases, and the rate of innovation at their firms.”

    By overcoming barriers to collaboration, app developers and security teams can leverage this knowledge to improve their approach and also come to the table with solutions aligned at moving up the stack. First, they can perform code analysis to uncover vulnerabilities early in the app development process. Second, they can work on stronger runtime protection once the SaaS application is deployed. Code analysis can be used to make runtime protection better, and runtime data can be used to improve code analysis. This creates a continuous improvement loop between the two and both teams can work on code analysis and runtime protection together.

    To achieve this continuous improvement loop, companies must work to break down traditional work silos that exist because of a mismatch in expectations. App developers, measured for speed, need to receive security information accurately, early and with sufficient context to correct problems during software development. Security pros can benefit by installing security tools in production that understand the security needs of their applications, as opposed to tools that require security pros to write rules, tune the tool and then deal with large number of alerts.

    The goal is to fix software mistakes, weaknesses and vulnerabilities at the beginning of the development process rather than after deployment, when liability, revenue and reputational risks are the greatest. It is also important to realize that achieving zero defects in source code is a noble but difficult-to-achieve goal. Instead of focusing on extremes, the goal should be to establish a continuous improvement logic in CI/CD (continuous integration/continuous deployment), where bugs are identified, some are fixed and the remainder have fast and accurate runtime protection.

    This shift in thinking and processes will give companies fewer security worries and enable them to more proactively embrace a SaaS-first mentality with higher confidence and trust levels. Companies will reap the financial benefits that come with strengthened industry reputations for rapid innovation capabilities and improved levels of software security.

    About the Authors

     Manish Gupta is founder and CEO of ShiftLeft Inc. He is a 20-year veteran of the security industry and has led product development, management and strategy at FireEye, Cisco Systems, Intel, McAfee and Redback Networks.

     

     

    Craig Rosen is vice president and CISO at AppDynamics. He has held leadership roles for more than two decades in the security industry, including at FireEye, Pacific Gas & Electric, TDI and CGI.

  • DevOps Chat: Developers and Security with Pete Chestna, Veracode

    DevOps Chat: Developers and Security with Pete Chestna, Veracode

    In this DevOps Chat we speak with Pete Chestna of Veracode about the roles and opinions of both developers and security pros about how and who should be working on security in the enterprise. Much of the discussion centers on a survey Veracode conducted with consulting firm Enterprise Strategy Group (ESG).

    Pete is always a great interview with a common-sense approach to DevSecOps borne from being both a developer and security pro. I am sure you will enjoy this discussion. As usual, the streaming audio of our discussion is immediately below and the transcript of our discussion is below that.

    Audio

     

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel of devopsy.kinsta.cloud here for another DevOps Chat. Today’s guest on the chat is Pete Chestna, Veracode, a CA company, director of developer engagement. And that’s a fancy way of saying Pete’s out there talking to developers about bringing security, shifting it left, bringing it into the developer universe and making all of our apps more secure. Pete, welcome to DevOps Chat.

    Pete Chestna: Thanks, Alan. Pleasure to be here.

    Shimel: Yep. So, Pete, you and I have, of course, crossed paths before and I’ve seen you present and you’re a knowledgeable, knowledgeable person, in terms of how security is continually kind of making inroads into the developer community and kinda convincing them that security’s everyone’s responsibility, right? That security isn’t just the purview of the security team; it’s everyone’s job. And you’re doing a heck of a job with it, which has, I think, attested to Veracode’s value in the market and its place in the market, so great job there.

    Chestna: Thanks.

    Shimel: Pete, what we’re gonna talk a little bit about today is that Veracode recently conducted a survey of developers and security folks, talking about this very subject. And I wanted to share it with our audience a little bit, if you can. Why don’t we start with what you consider sort of the headline of this survey?

    Chestna: Thanks. So this was done with the enterprise strategy group and the key takeaway for this is, really, again, it’s reinforcing the messages we already bring out to the market, so we need increased accountability. The reason that developers don’t take security more seriously is because they’re held to account for their functionality and not necessarily their security, so, if we bring in metrics and goals into the equation, for developers, we’re gonna see an increased uptick in the way they think about security. It’s about building relationships with your security team, so security and developers need to work together to build that relationship, so, that way, they feel like there are people on each side and not just a function.

    And, really, it’s what I call the “three-legged barstool of application security.” It’s about training, making sure developers are trained—a lot of the times that they don’t wanna develop these tools is because they don’t know what to do with the results. Second is integrate and automate—if you bring it into their toolchain and make it easy for them to use, then they’re more likely to use it. And, lastly, it’s about helping them fix what they find—again, this knowledge gap of, “What is across-site scripting? What is OS command injection?” Help them understand how to fix these and prevent them makes them more likely to adopt these tools and make security a better part of their work.

    Shimel: Got it. And you know what, Pete? So I’m up here in Boston, recording this, and we just finished a DevOps Connect. I had an interesting question from the audience yesterday, and a gentleman said, “Well, our developers kinda get paid or judged on how much code they commit. And to go back and tell them that the code they committed, you know, it had vulnerabilities and they gotta fix it, it really upsets them.” I’m sorry for them, right, Pete? But –

    Chestna: Right.

    Shimel: – to me, it’s kind of—you know, I used to see this problem with running sales teams and stuff like that. You’ve mismatched incentives here. Right?

    Chestna: Oh, absolutely.

    Shimel: You know, and that has to be—and that’s not the developer’s or the security guy’s fault; that’s management’s fault.

    Chestna: Yeah, if we don’t take accountability for what we’re doing, if we don’t hold the developers—so, you know, I always look at the security team. They are responsible for security but the developers need to feel accountable for security. And you’re right; it’s management that needs to say, “We’re gonna measure you on this and it’s gonna be part of how we evaluate you at the end of the year.” That will drive the right behaviors. They’re gonna do what they get paid to do. And if they get paid to produce functionality faster, that’s exactly what they end up doing.

    Shimel: Yep. And so this is apparently something, I think, we need to solve. You know, we’re gonna see real progress in the area. But, look, let’s save that for another day. Pete, can you share some more of the findings on the survey with us?

    Chestna: Yeah, absolutely. So a couple of things that stood out to me, as I read through it, was there’s a increased maturity as far as adoption of Agile and DevOps. There’s only 5 percent of respondents that said they have no Agile in their shop at all, where 18 percent say that most or all of their products are using a Agile methodology. And then it goes on from there—28 percent is more than half. So we’re seeing a really good adoption, at least of Agile. And then if you think of DevOps usually being run on an Agile methodology, only 6 percent of respondents says they haven’t adopted DevOps at all and 17 percent are reporting that they have extensively adopted it, so great numbers, as far as people coming up to speed in doing this. And, as far as making things easier, 45 percent of respondents said that DevOps makes it easier for them to developer software, including security, which is a great finding.

    Shimel: That is—what was the number? 45? Just about half?

    Chestna: Yeah.

    Shimel: Excellent.

    Chestna: So those must be shops that are doing it well because they’re seeing the benefits that you get from experiencing DevOps.

    Shimel: Well, I mean, you know, Pete, you take this, I think, in conjunction with the recent release—I was out in London last week for DOES and the folks at DORA and Puppet released the State of DevOps for this year’s survey and results, and it dovetails nicely here, where DevOps-enabled organizations deal with something—I forget what the number was—15 percent less vulnerabilities or 15 –

    Chestna: Right.

    Shimel: It was some –

    Chestna: It was from the Puppet report.

    Shimel: – eye-popping number of the improvement in the quality of code, vis á vis security. So it seems to make sense, but you know what, Pete? I’m amazed that something that seems so, you know, simple, so elementary, if you will, is still—we’re still talking about only half of organizations, a little less than half organizations doing it. What do you think is holding back the other half?

    Chestna: It’s the culture. You know, people wanna maintain their silos. I see it all the time where the management—again, we’re going back to management, as a problem here—they wanna hold on to their silos and their power. And if you look at where DevOps takes a company typically, if you do it whole-hog and embrace everything, you’re reorganizing, so someone that has a large quality staff reporting to them is now gonna have that broken up and disseminated into other teams, and now they’re gonna be responsible for other functions or they’re gonna have to go find another job. So there’s this big unknown and fear for them of, “What does the future hold if I just don’t do quality and I have to actually produce product?” So this cross-accountability is just—is scaring the hell out of them.

    Shimel: Yeah. Yeah. But you know what? Let’s not put it all on the dev side of things; let’s talk a little bit about the security folks, too. And, Pete, you and I both come from the security world. There are a lot of security people who get very territorial, you know, to say the least, around running scans, being the ultimate deciders of what’s secure and not secure, and they’re just as resistant to change, from a cultural point of view, as dev teams are sometimes. I mean, do you agree or –

    Chestna: Yeah, and they white-knuckle the release, so –

    Shimel: Yeah. [Laughs]

    Chestna: – I’ve talked to a lot of people that say, “Hey, it’s gotta be pen-tested. I have to pen-test everything.” If you’re talking about releasing multiple times a day, there’s no way in heck that you’re gonna be able to do that. So you have to pick your battles and say, “This is important. It touches crypto or this touches authentication or authorization. It has to be pen-tested,” versus, “I can use the easy, least expensive way of SAST and DAST to go and find vulnerabilities and I can trust that those tools will find things.” So it’s that back-and-forth of how much security needs to be in there and how assured do they need to feel before they can sleep at night.

    Shimel: Mm-hmm. And they’re ultimately, Pete—you know, I think, when we look at surveys like this and we talk about the view of the security team versus the view of the dev team, view of management, really, the way to kinda bridge these gaps and bring this all together is really a lot of the kinds of things that you do. Right? You’re out there, talking to the different constituencies around the world, almost constantly—we were talking off-mic about how much both of us have been traveling—but there’s a mission there to bring these tribes together, if you will, and forge a common path and a common way of working together. And I think, if we take surveys like this and we look at the results and we share those results with the two teams, again, it becomes sort of like, “Duh. What are we fighting about?”

    Chestna: Oh, exactly. So, I mean, if I go back to the survey for a second here and look at how people are being measured, 33 percent of respondents say, “I’m being measured on functionality that I released,” versus only 18 percent that are saying, “I’m being judged on security.” So, if you look at that number and say, “Well, what are they doing? Their actions reflect how they’re being measured, so let’s change how they’re being measured and see if that changes what they think about their jobs.” The people that responded and looked at “Does DevOps help me with security, the ability to automate and integrate?” 58 percent of respondents said, “Hey, it allows me to do this integration and automation,” which is one step toward getting them to actually pay attention to those results.

    Shimel: Yeah. And 58 percent is a sizable majority.

    Chestna: Yes.

    Shimel: That is Pete, I don’t know if you have the insight on this, but these were large enterprises, primarily, that were surveyed here?

    Chestna: Yes, most of them were—98 percent were 1,000 or more employees.

    Shimel: Okay. Now were there any particular verticals that were highlighted or anything?

    Chestna: The biggest ones were IT at 23 percent and the finance industry at 21 percent.

    Shimel: Oh, okay, there you go. And those are both pretty DevOps-savvy, if you will, or maybe early—no, I don’t wanna use the term “early adopters,” ’cause finance is very rarely an early adopter, but, I mean, certainly, DevOps has taken a strong foothold within the financial and financial services industry, so.

    Chestna: Yeah, and the thing is driving them is really the regulatory compliance, so 53 percent of respondents said that, “We’re doing app-sec because of regulatory.” Forty percent are saying it’s because of corporate governance. And an interesting stat here was 34 percent said it was because of customers or partners, so the great news is people are starting to pay attention to their software supply chain, to say, “Am I getting secure software to install in my environment?” and they’re asking customers to prove that they are. And, hopefully, that also means that outside customers, like individuals, are also looking at security as an important decision in where they go and where they spend their money.

    Shimel: Yep. I agree. I mean, it is interesting stuff, Pete. So, Pete, where do we go from here with this now?

    Chestna: So I think the next step is to continue the journey that you and I are on, of helping security understand development, help development understand security, tell them that they need to build in accountability. It’s built by relationships, so they have to work together. If security starts to teach developers how to do secure development and explains to them how to fix the things that they’re finding, developers are more likely to engage because it’s not this great unknown of “Hey, I don’t wanna go in and fail. I don’t wanna go in and look like an idiot,” or, “I don’t wanna go in and not understand what I’m doing.”

    So security needs to take their part and companies need to understand that training and development and education around security is a carrying cost of having developers. They were never trained to do it and they have to pick that cost up; it’s not like they can just throw them all away and bring in new developers because the developers that they bring in are gonna have the same problem. So getting that education done leads to better engagement, leads to better results.

    Shimel: Agreed. And you know what, Pete? Just a shameless plug on that note: I’m gonna be out in Singapore at the end of July, at the RSA APJ show, putting on another DevSecOps event at RSA, and we’re working with both the local—so it’s interesting. In Singapore, there’s almost a 900-person-strong DevSecOps MeetUp group, which is huge for that area of the world, and there’s an equally-as-big DevOps MeetUp group. And we’re working with both of these MeetUp groups, as part of this event, and we’re really—I’m hoping to really kinda bring these tribes together and make it happen. Of course, I’d –

    Chestna: Sounds great.

    Shimel: – you to come out. You’re in Black Hat at that time.

    Chestna: Yes.

    Shimel: Are you speaking at Black Hat, Peter?

    Chestna: I’m not. I’m doing customer stuff, customer-facing.

    Shimel: Ah, very good. So you’ll be in hot Vegas and I hear Singapore’s pretty warm now that time of year as well. Well, we continue along our mission of spreading the good word, right?

    Chestna: Yes, sir.

    Shimel: as they say. Anyway, so, Pete, we’re about out of time. I know you don’t have the URLs off the top of your head right now, but we will include them in the show notes for people to download the results of the survey and the report, maybe find out a little bit more, but I wanted to thank you for sharing and giving us a little bit of insight into the results. And then keep up the great work, Pete. You do good stuff.

    Chestna: Thanks. I appreciate your time, Alan.

    Shimel: All right, Peter. This is Alan Shimel for devopsy.kinsta.cloud’s DevOps Chat and we’ll see you on the next chat.

    — Alan Shimel