devopsy.kinsta.cloud

Author: Walter Capitani

  • How Third-Party Security Assurance Enhances DevSecOps

    How Third-Party Security Assurance Enhances DevSecOps

    Enterprises are constantly trying to do more with less today, and do it faster to gain competitive advantages and grow revenue. Nowhere is this more prevalent than in their internal software development processes. The movement to third-party or external sources of code is a natural reaction to “faster” release cycles. However, this need for speed can have unintended downstream security consequences for application development teams.

    As cybercrime has grown parallel to the explosion of digital processes in many enterprises, “security by design” needs to go from a nice-to-have concept to a hard reality. This is the reason DevSecOps has gone from buzzword to practice for many development teams. Implementing a software security assurance program, which embeds security as part of the coding and assembly process rather than a separate, after-the-fact quality testing function performed by security staff, is vital to limiting the cost of fixing vulnerabilities and preventing data breaches.

    Software Attacks are a Growing Problem

    One of the recent and most high-profile cybercrime incidents involved hackers using the SolarWinds software platform as a launching pad to breach companies. With threats specifically targeting software on a growth curve, organizations need to secure both the software they develop and that which they acquire from suppliers. This often includes a lot of reused code picked up from vendors and open source libraries that offer the building blocks for software and applications to help developers work faster and cheaper—but with a potential security cost.

    Maintaining software security has become more complicated as enterprises rely more on third parties for the components to develop software faster and cheaper and keep up with digital disruption. The share of code developed in-house for enterprise software has dropped from 43% to 38% in the last five years, while the use of commercial and open source software has grown in its place. Rather than rely on custom code, many developers are leveraging open source components or API libraries to build applications. Some software vendors provide application libraries that can have third-party components purchased by that vendor for their customers.

    Leaning on third-party code can leave software open to abuse, as developers have less visibility into the content of the components they use as the building blocks of their programming. Some vendor libraries get reused by the vendor for different applications, and some use open source components in their code; a developer may be using third- and fourth-party code from who-knows-what source. Vulnerabilities can be hiding many layers deep in the code, and unless developers know the source, they are difficult to detect.

    This is where a well-established software security assurance program can reduce the risk of cybersecurity incidents and the impact of security weaknesses. It protects users and the enterprise by establishing cybersecurity practices, fulfills the organization’s risk management and compliance requirements and offers the C-suite visibility into the process.

    A Checklist for Software Security Assurance

    A well-organized security assurance program ensures that security requirements have been established for the software and the software development process and that any software deployed is free from vulnerabilities. If any are found, it also establishes remediation steps to handle them.

    Some important practices:

    ● Trust but verify: Ensure that a security evaluation has been performed for the software throughout the software development life cycle (SDLC). Proactively validate the security and integrity of software code and provide results identifying any vulnerabilities found and assign remediation steps to correct them before release into production.
    ● Know what’s in the software: Since many developers and software vendors rely on open source and third-party code more frequently, it’s more important than ever to know what those components are and where they come from. Software security
    assurance should include the software bill of materials (SBOM) for any open source code and third-party components.
    ● Put it in writing: Produce documented proof, such as the SBOM mentioned above, and generate a vulnerability report proving the integrity of the software. If vulnerabilities are found, determine the cybersecurity risk and the potential impact based on the Common Vulnerability Scoring System (CVSS) score, an industry standard, to guide remediation.

    A good software security assurance program makes sure that each software review or audit includes an evaluation of the security requirements for the enterprise. These can vary by industry—and even within the organization—based on which function the software is running. A program running mailing lists for a power generation company is not as mission-critical as the software running the power grid.

    With all industries in the crosshairs of cyberattackers, an effective and diligent software security assurance program will make it harder for them to find a back door and can help keep your organization safe.

  • Prevent False Positives From Derailing Shift Left

    Prevent False Positives From Derailing Shift Left

    Static application security testing (SAST) tools are designed to balance false positives (incorrect warnings) with false negatives (missed vulnerabilities) primarily because deeper analysis requires more time and computing resources. Both of these are in short supply among developers that are tasked with meeting shorter and shorter product delivery milestones.

    So, while SAST vendors consider a true positive a correct detection of a real defect, and a false positive an alert on a bug that does not exist, this is rarely the lens used by developers. What really matters to them is whether the output of a SAST tool is useful and actionable. For example, there is a great deal of variation in how developers interpret SAST scan results, depending on the nature of the defect, the role of the user, the platform on which the application will run and the environment in which it is deployed.

    Take, for example, a true positive result of a buffer overrun, one of the most notorious classes of C/C++ defects from a security perspective. In the early stages of application development, it almost always makes sense to change the code to fix such a bug. Developers are actively changing the code, anyway, so fixing it involves little extra overhead. However, if the same defect is found after the application has been deployed, then it is much trickier to decide whether it is worth fixing.

    That’s because the cost of fixing a real bug may exceed the benefit of fixing it, and the benefit of “correcting” a false positive may exceed the cost of leaving it alone. Since SAST tools can only be relied on to give narrow technical answers, humans must interpret and determine which static analysis results should be acted upon.

    Consider a hypothetical comparison of different SAST tools in the graph below. Tool A has good recall (i.e., the ability to identify real defects) and precision (i.e., the ability to exclude false positives) which results in finding many of the real bugs with a reasonable amount of false positives. Tool B has high precision but poor recall, resulting in low false positives but a higher number of false negatives (undetected security vulnerabilities). Tool C has poor precision but high recall, resulting in detecting all the possible errors but also with a very high number of false positives.

    A comparison of recall and precision of hypothetical SAST tools.

    Developers hate false positives because they introduce unnecessary work and delays. This, in turn, has a disproportionate effect on the way tools are designed, configured and used. If given a choice between Tool A that reports 40 real defects and 10 false positives, and Tool B that reports 50 real bugs but with 50 false positives, users will almost always prefer the former, even though it is finding fewer real defects. This is perfectly understandable — users are asked to weigh an immediate concrete negative (time spent looking at false positives) against an intangible potential future positive (vulnerabilities that may not show up).

    However, if one offsets the time and risk saved in finding those 10 bugs earlier (i.e. by avoiding expensive and potentially dangerous bugs in finished products) against the time needed to assess the additional 40 warnings as false positives, then it quickly becomes apparent that configuration B is more economical. This is especially true with security vulnerabilities that slip through testing and can have expensive consequences.

    Since the number of warnings within a DevOps pipeline can be a deterrent to getting the most out of a SAST tool – especially early in the project, when it provides the greatest benefits – organizations should consider the following techniques to increase developer attention to alerts:

    • Filter and Focus: Parse the viewed data from the tool’s interface, focusing on what is most important for the project and assigning developers to fix critical issues in priority order.
    • Mark and Defer: Lower the priority, or change the state to “later,” for example, on all or a subset of the warnings based on some set of conditions that are less crucial to the project.
    • Stop the Bleeding: Use the above techniques to temporarily defer existing warnings with the emphasis on fixing new defects that are introduced as code is changed or new code is developed.

    Despite the fact that it will invariably produce some false positives, SAST helps shift security further to the left by ensuring security and programming vulnerabilities are eliminated when code is being written, rather than at the testing stage when they are much more expensive to fix. Filtering and prioritizing warnings are two effective techniques to ensure developers don’t throw out the baby with the bathwater.

  • A Pragmatic Approach to DevSecOps

    A Pragmatic Approach to DevSecOps

    In response to market and competitive forces, many companies are using DevSecOps to develop higher quality and more secure software. Whether that software is their own product or is a component of their product, quality – and especially security – have become required rather than optional. To accommodate the security requirements, some companies have adopted DecSecOps overlays to their software development process that can, and has, caused significant disruption.

    Barriers to DevSecOps Adoption

    The main barrier to the transition from DevOps to DevSecOps is organizational rather than technical. Many companies are already employing special security teams tasked with ensuring applications are secure and that policies are being followed. However, they tend to act as gatekeepers to the release process – nothing gets released without security testing. This approach is prone to be “too little, too late,” as the security team is usually involved near the end of the development life cycle. Attempts to “test in” security rarely succeeds. Security needs to be built in from the start; “shifted left” so that security is fully integrated into the process from the start. Let’s break down these barriers:

    • Lack of motivation and awareness:  There’s a lack of awareness about security issues, and about the risks and impacts security vulnerabilities pose to the organization in terms of liability and reputation. Vulnerability statistics make it clear that many companies are still not taking security seriously enough. This indifference remains, even when 63% of companies had, or nearly had, vital data breaches in 2019 and each breach cost an average of $3.9 million to deal with.
    • Distrust: Lack of trust between teams is expected in most organizations, but it’s particularly problematic between software developers and security teams. Developers view security as impeding their success, and security teams view developers as incapable of delivering secure software.
    • Poor Perception: The prevailing view of security is that it’s difficult to implement and somebody else’s problem. If security teams are solely responsible for security, then it’s easy for developers to delegate to them. Companies often think that security can be tested into the product at the end of development – with disastrous results. Vulnerabilities discovered at this late stage are more difficult to debug, with subsequent impacts on time-to-market, budgets, product quality and reputation.
    • Gatekeeping: The security team sometimes acts as gatekeepers of release candidates, which is a source of friction between developers, testers and security experts. Placing security as a last, QA-like step before delivering a release is too little, too late. Developers also don’t like other teams imposing restrictions on their work, which is an issue when trying to put security controls, like secure coding practices, in place.
    • Lack of integration: Without integrating security practices and tools through the entire development process, security improvement initiatives have trouble getting off the ground. In fact, the key tenet of DevSecOps is to integrate security practices into each phase of development.

    These barriers might seem daunting, but it’s not unlike other corporate cultural shifts. Security teams need to delegate aspects of security to development teams and integrate secure practices as early as they can in their work. Developers need training and understanding of how critical security is to their work. Developers already have an appreciation for safety and quality, but lack some of the basics of good security practices.

    It doesn’t need to be this way. However, removing these barriers won’t happen overnight, so a practical and gradual approach to improving security is needed.

    Easing Your Way into DevSecOps

    Transitioning to a more secure software development process across an organization rarely works when done with a “big bang” approach. There’s often lack of buy-in from the teams, and general resistance to change when new approaches haven’t proven themselves. The cultural barriers to adoption remain, and the initiative fails. As with any large task, it makes sense to start small; to learn and adapt before expanding to more teams. The move from DevOps to DevSecOps should be organic.

    Start Small

    The alternative to the doomed “big bang” approach is to start with a small, motivated team, where it’s possible to implement new practices from the very beginning of a project. By keeping the scope, and thus, the technical and business risk, of the project smaller, it is possible to allow time for learning and adjustments along the way. Think of this as an incubator project, where the results are analyzed and used to evaluate gains, losses, what went right and what went wrong. This team can learn, and then teach the larger organization what works and what doesn’t. Importantly, they set the standard for how to approach DevSecOps, and kick off the organic growth needed for a smoother transition.

    Starting small extends to the use of new tools, as well. It’s one thing for a small team to adopt new technologies, but they can be quickly overwhelmed. Static application security testing (SAST) tools can produce a lot of information. It’s important that initial projects concentrate on high impact, high probability vulnerabilities, first and foremost. It’s easy to get bogged down in the details, and discipline is needed to focus on the highest impact, and to get the most ROI out of new tools.

    Start at the Beginning (Or as Early as you Can)

    This is where “shift left” plays an important role in the transition to DevSecOps. A small team can introduce new techniques without a large initial impact on productivity. They can also start integrating security from the beginning, right at the concept phase of development. They can introduce new tools, and become proficient at using them. It’s at these early stages where the payoff is the greatest. Finding and fixing possible vulnerabilities, and adopting more secure design and coding practices, greatly reduces the downstream impact. It’s not hard to justify the effort, when security attacks and data breaches in production systems can cost millions in fixes, downtime and reputation.

    Measure Success

    The interesting thing about measuring DevSecOps success is looking at what isn’t there; security vulnerabilities and potential software weaknesses. To understand how well a new DevSecOps transition is working, it’s important to have a baseline of vulnerabilities that exist in your other applications. For example, how many vulnerabilities were discovered during security testing? How many vulnerabilities made it into shipping products? If these statistics are known, then it’s a case of measuring how well the new project does versus the baseline performance.

    Other metrics are important, too, such as measuring the number of vulnerabilities detected, identifying those that are true risks and those that were removed through corrective action. SAST tools provide information on a per-build basis, so it is possible to look at trends over time. You can evaluate the DevSecOps process in terms of vulnerability trends over time; are they converging to a goal? How quickly is the convergence?

    The ultimate measurement of success is a more secure product and, ideally, security testing at the later stages of development should find fewer vulnerabilities than in the past. After building in security and starting at the beginning with a shift-left approach, vulnerabilities are caught sooner and removed more quickly.

    Start your DevSecOps Transition

    The critical time to detect new security vulnerabilities is as soon as developers write new code (or test cases) before it’s submitted to a build or software control system. It’s also critical that tools work in your favorite development environment (be that VI, Emacs, Microsoft VS Code or Visual Studio, or any of a number of Eclipse-based environments). It’s at this point where SAST tools shift security improvement via automation to the earliest point in the development cycle. Finding and fixing vulnerabilities here is less risky, cheaper and easier.

    Equally critical is establishing an initial baseline inventory of your third-party software assets, whether that is open source, reused internal code from other projects and/or third-party commercial software. This is done at the very beginning of the project, about as shift left as it gets! Software composition analysis (SCA) is critical throughout the development process since the open source and third-party software that you depend on is likely to have updates and introduce new security threats over time (including after you release the product).

    Early adoption of SAST and SCA tools can have many benefits as part of your DevSecOps adoption. These tools used in conjunction with a security “built in” approach make this transition easier:

    • Improve security awareness by exposing vulnerabilities in existing code, third party software and newly developed code. SAST tools provide all the necessary information to fix potential vulnerabilities, but also provide an explanation of the severity and risk. Developers get more experience with the types of software weaknesses that lead to vulnerabilities, and learn to avoid them.
    • Illustrate ROI by measuring success, and have a goal in mind. Eliminating just one serious vulnerability justifies much of the investment. However, communicating this ROI is important, and needs to be put in the proper context of how costly and risky vulnerabilities are once a product is released.
    • Remove gatekeeping so that developers and the security team are working together. Security teams provide the security controls that developers need to follow with a “trust, then check” approach. Developers learn more about secure practices over time, and integrate that knowledge into their existing workflow. In the end, there is no security team gate at release, and all members of the team know and understand the state of the product.
    • Integrate security practices into software development gradually, starting with a small team and small project. As experience is gained, these teams train the rest of the organization on best practices.

    DevSecOps From the Ground Up

    DevSecOps is a way to build security into an application, and the transition from an existing CI/CD and DevOps process is made easier with the right approach. Successful transition depends on breaking down the resistance to security practices that exist in many organizations. A “start small and start early” approach is recommended where new techniques, processes and tools, such as SAST and SCA, can be integrated with less downtime and cost.

    Most importantly, you need to decide what the success factors are. Measuring success and the ROI of the effort is needed to grow DevSecOps organically within the organization. Tools and automation provide the data needed to show progress, but communicating the ROI needs to be done in business terms. In the end, success results in a more secure product, and reduces your company’s exposure to data breaches and cyberattacks.