devopsy.kinsta.cloud

Category: IT Security

  • How Log4j Becomes a Serious DevOps Problem

    How Log4j Becomes a Serious DevOps Problem

    The recent discovery of the Apache Log4j vulnerability has wide-ranging implications for anyone who develops software, especially for those in the DevOps realm. What’s most troubling about the vulnerability (CVE-2021-44228) is how prevalent the use of Log4j is. The vulnerability is reported in a vast array of applications and directly impacts numerous Apache projects, including Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark and Struts. There are also numerous Cisco programs impacted by the vulnerability. The list of impacted applications and libraries seems to be almost infinite, extending to Elastic LogStash, GrayLog2, Neo4J, Steam, Twitter and even the VMware Tanzu family of programs. 

    Simply put, any application that uses Log4J for logging and tracing is subject to the identified family of attacks known as Log4Shell. Those attacks prove easy to exploit and can be used to take control of vulnerable servers.

    Why DevOps Practitioners Should be Concerned

    The world of DevOps is all about reiteration, where features and improvements to applications are slipstreamed into production environments. To accelerate development, the QA process is usually limited to the most recent changes.

    In other words, developers, via their pipelines, can often establish a baseline of quality and then speed up their CI/CD pipelines by testing only what has changed since the last iteration of the application. This process ensures an acceptable level of quality without introducing what are perceived to be unnecessary steps.

    However, those processes do not take into account newly discovered flaws in older APIs and libraries, meaning that those flaws could telegraph into deployed applications and leave DevOps teams completely unaware of the impact and possible disruption. Detecting flaws after an application is deployed is like putting the cart in front of the horse.

    Further complicating the situation is that many developers lack a well-defined and well-maintained software bill of materials (SBOM), which means developers may be unaware of the components and/or libraries being used in a deployed application.

    Without an SBOM, it becomes increasingly difficult to remediate an issue with an application—especially if developers are unaware of the issue to begin with.

    Where Should Responsibility Lie?

    More often than not, vulnerabilities introduced into applications become a game of finger-pointing, where developers blame library suppliers or infosec staffers, and infosec blames developers for not performing due diligence on the components they are selecting to incorporate into their applications.

    Of course, finger-pointing only adds to the angst and delays the process of remediation and does nothing to determine where the responsibility for detecting and remediating vulnerabilities lies. DevOps teams need to put these squabbles aside and establish a process that deals with unintentionally introduced vulnerabilities. What’s more, DevOps teams need to invest in DevSecOps to make sure that vulnerabilities are detected and remediated as quickly as possible.

    To avoid the potentially massive disruption that a vulnerability like Log4Shell can cause, those running DevOps should consider:

    • Creating and maintaining an SBOM 
    • Deploying automated tools that scan for discovered vulnerabilities
    • Automatically comparing the SBOM against vulnerabilities
    • Including DevSecOps processes within their CI/CD pipelines
    • Frequently testing applications for flaws
    • Coordinating with cybersecurity personnel when a flaw is discovered
    • Patching frequently

    With more and more vulnerabilities discovered on an almost daily basis, DevOps teams must be ever vigilant about what components are incorporated into their applications. What’s more, with the increased use of low-code/no-code tools and integrated development environments (IDEs), DevOps will also need to consider if those tools use other libraries that can become compromised.

     

     

  • Bridging the AppSec and DevOps Disconnect

    Bridging the AppSec and DevOps Disconnect

    Research estimates that cybercrime is going to cost the world $10.5 trillion annually by 2025, so it is no surprise that cybersecurity has become a top priority for business leaders. 

    Today, security teams are striving to harden their systems against cyberattacks and improve their resilience through more employee security training, incident response programs and the implementation of technology to monitor and detect threats. 

    Many organizations are also adopting a shift left security strategy, where security is being addressed earlier in the application development life cycle. These organizations are realizing that finding security issues in their software after it’s released is much riskier and more costly. After all, the longer a bug is in the wild, the longer an attacker has to find and exploit it. By implementing a shift left security strategy, the window of opportunity for an attacker decreases or can be completely removed if a security issue is addressed in the development stages, while also offering organizations significant security benefits and cost savings. 

    Shifting left is a security testing strategy where DevOps and security teams work in tandem to identify and address vulnerabilities during early development stages. This process significantly improves security as it means vulnerabilities are identified and mitigated before they are introduced to the enterprise environment.

    However, one of the biggest challenges organizations face when adopting a shift left security strategy (DevSecOps) is encouraging security teams and DevOps teams to work together.

    Encouraging AppSec and DevOps Collaboration

    Until recently, DevOps teams and AppSec teams were completely siloed. DevOps teams worked on accelerating the speed at which applications are released into the enterprise; application security teams only got involved in the later stages. However, this process has caused security concerns, particularly as more organizations move to agile development. 

    When security teams eventually got their hands on new applications and updates, security flaws were inevitably found that caused delays and friction with DevOps teams. 

    However, as organizations begin to embrace DevSecOps and shift left strategies, these two previously siloed teams are being forced to work together more closely. So, how can the friction be reduced and the disconnect bridged?

    Culturally, some ingrained attitudes and behaviors challenge the success of any DevSecOps efforts. Security teams have seen DevOps processes accelerate the speed at which software is delivered, but without security considerations, while DevOps teams experienced security slowing down processes and giving inconsistent results and feedback on security issues. Each party has their own manager to please; their own set of metrics that they’re measured against and a priority list as long as their arms already. Both teams follow different processes and, crucially, use different tools. DevOps can’t get around the security tool complexity and lack of integration with their existing toolset and security teams have no control over the CI pipeline to best implement security assurance.

    One of the best ways to overcome this friction is through better technology, process and culture that enables collaboration between teams. First, DevOps teams do care about security, but it might be lower on their priority list. Security teams must understand that DevOps teams care about code, quality and efficiency. They are not security experts and just because an organization is shifting left doesn’t mean they need to become one. Instead, organizations should implement tools for DevOps where security checks can be easily embedded.

    So, what can organizations do to improve the collaboration between AppSec and DevOps teams to achieve DevSecOps?

    DevSecOps is a culture and it requires a shift in thinking. Security teams and DevOps teams need to understand each other’s priorities and understand that by working together they are improving the overall security of the organization and decreasing the risk of cyberattacks. Does the DevOps team understand the potential risks a vulnerability could have on the organization overall? If not, security teams need to educate them.  

    Security automation tools offer great support when organizations are moving to DevSecOps, as they allow bugs and vulnerabilities to be flagged automatically as applications are being developed. This will allow them to be addressed before services go live; however, because they generate frequent alerts, this could add to DevOps’ already busy workloads. By deploying tools which can be synced with CI/CD pipelines and provide custom advisories so DevOps teams can address bugs themselves without having to ask AppSec teams, this can significantly speed up the process, reduce friction and ease workloads.

  • Log4j: It’s All About the Supply Chain, Baby!

    Log4j: It’s All About the Supply Chain, Baby!

    In 2021, the security story in DevOps and DevSecOps has been the supply chain. So, it’s only fitting that we are currently experiencing the mother of all supply chain issues with the Log4j Log4Shell RCE vulnerability to close out the year.

    I won’t waste your time rehashing what Log4j is, why it’s so dangerous and warning you to test all of your own and third-party apps to make sure you’re insulated against this. I will say, though, that I don’t think Log4j is a bug—I think it is a defect in design. Worse, as we are seeing, there are multiple defects in design. All of you Java haters out there—back in the box. Java is no worse or better than any other language that we regularly use to build applications.

    So, what exactly is so special about supply chain security? Why do DevSecOps teams, developers and our security incident management teams have to interpret this as a warning and stay at the top of their game from now on? It’s simple. In the age of SaaSOps; in a world where we Frankenstein applications by cobbling and stitching together numerous third-party components (some open source, some not), we are increasingly reliant on code, even in our own applications, that we did not write. We probably didn’t test it, either. We rely more than ever on the “community” and on unknown third-party suppliers. It has always been a leap of faith with these apps and components, and this year some of those chickens have come home to roost.

    The White House executive order on improving the nation’s cybersecurity calls for a software bill of materials (SBOM) for applications; that is, as they say at the tables in Las Vegas, a fine start, but we need more. If we really had a BOM included with the apps we use (and that includes SaaS apps), how many of you are actually going to read it and act on that information? And besides including all these components in a BOM, what further obligations do suppliers have to ensure that they are replacing and updating vulnerable code?

    This issue becomes increasingly acute with open source. My partner and colleague Mitchell Ashley recently asked if too much open source was a bad thing. I don’t know if I would go that far; the more the better, as far as I am concerned. But we do need better monitoring and management of the open source code that’s out there. More testing before it goes GA, maybe? Yeah, the Cadillac open source projects might have the money and time to do that, maybe, but take Log4j as a perfect example: It’s two guys maintaining it in their free time and trying to do right by the community. How can they possibly keep up?

    I used to think that the answer was in the repos. My friends at Sonatype had a component firewall that would not let you download an old or vulnerable version of code without warning. That was fine if you were downloading it for the first time. But if it was already embedded—then what? Can we scan for older components in an app? My friends at JFrog can do something similar and more with Xray. And Sonatype’s Nexus platform also allows current and prospective users to scan older apps for vulnerabilities—including scanning for Log4j, absolutely free. But is that enough?

    All of these questions involve open source components. What about third-party SaaS apps? Often, we don’t even know what components or code are in those SaaS apps. Again, it is a leap of faith we are making when using them. 

    These supply chain security issues are a problem that we are just beginning to grasp. And while 2021 has been the year the stuff hit the fan and Log4j may be remembered as the biggest of the first, I, for one, am afraid to think about what else is coming. 

    I do know that the only way to tackle this and any future supply chain issues is if developers, DevSecOps, DevOps, SREs and our security incident response teams all work together. Just like most security issues, supply chain security is all of our responsibility.  I hope Log4j hasn’t bitten you too hard, and that you are on top of your exposure. But let’s start thinking about the next one and how we can better prepare for it.

  • Log4j: Is There Such a Thing as ‘Too Much’ Open Source?

    Log4j: Is There Such a Thing as ‘Too Much’ Open Source?

    The Log4j vulnerability got me thinking: Is there such a thing as too much open source?

    Before anyone immediately fires off a flaming email, rage tweet or scathing blog post, hear me out for a moment. If you know me, you know that I am an open source fanatic. I’ve been asked many times, “Should we use open source software and, if so, how much?” My response was (and still is) that open source software is a massive source of innovation and I can promise you, whether or not you use it, know that your competitors certainly are!

    I could go on and on extolling the virtues of open source and how it enabled the cloud, reshaped software architectures, created new product categories and gave us valuable tools like telemetry and increased visibility into cloud-native apps, infrastructure and beyond.

    When I first heard about the Log4j vulnerability on December 9, like many others, I quickly informed the tech leaders, engineers and SREs in my network about the serious potential risks such a zero-day vulnerability in the widely-used Log4j tool represented. In one way—probably the only way—the Log4j exploit shared a very important characteristic with the SolarWinds supply chain attack of late 2020: The widespread distribution and ubiquitous use of the software across applications, systems and organizations. Arguably, Log4j is used much more widely.

    This similarity raised questions I hadn’t fully considered in the context of such widely used software. Can we use too much open source software? Is Log4j too prevalent in its use and, therefore, a single point of failure? How bad would the Log4j situation be if a fix wasn’t immediately available or if one wasn’t forthcoming?

    The answer I kept coming back to is a resounding ‘no’. I still don’t think there’s such a thing as using too much (insert any open source project name) software. Log4j—and so many other open source software projects—are massively beneficial to us, and we virtually couldn’t function without these systems, tools and solutions in today’s age of ubiquitous software.

    What the current Log4j vulnerability, recent software supply chain attacks and other cybersecurity incidents with widespread impact highlight is our need to quickly respond to security incidents and take remediation steps anywhere they are found in the software stack. And that includes open source software. High-impact security vulnerabilities can occur anywhere across the numerous software stacks and services we use, though they have even greater impact when they occur in widely-used infrastructure, reusable components and microservices.

    In a world of infrastructure-as-code (IaC), it’s no longer a matter of patching a particular set of routers or servers. Unfortunately, much of the software stack is not containerized or built using smaller, easily replaceable containers and microservices—at least not yet. Our mindset and processes going forward from this incident must change, and we have to be prepared to respond to and remediate vulnerabilities both horizontally and vertically across the software stack. In certain circumstances, we have to do this on a large scale.

    As is common with active open source projects, Log4j contributors responded quickly with a readily available fix. The real answer to my original question above is that few are as prepared to respond to an important security vulnerability as those incredible, hard-working, active and engaged open source software developers.

  • Log4j Puts Effective IT Operations at Center Stage

    Log4j Puts Effective IT Operations at Center Stage

    News of the Apache Log4j vulnerability exploit is striking fear into the hearts of both software makers and users. Log4j is the most popular Java logging service used today, with over 400,000 GitHub downloads and has been embedded in most internet services and products from companies all over the world, including Apple, Amazon, Cloudflare, Steam, Tesla, Twitter and Baidu, just to name a few.

    Exploiting this vulnerability is simple and allows threat actors to control Java-based web servers and launch remote code execution (RCE) attacks. New variations of the original exploit are being introduced rapidly—over 60 in less than 24 hours. The attack surface grows by the minute. Good credential management and advanced authentication will not help as this exploit is pre-authentication, which means an attacker doesn’t need to authenticate to your web applications to attack. Your web server is vulnerable.

    The good news is your organization likely has a team of seasoned IT operations pros that can help protect against the worst of Log4j’s devastation.

    How IT Operations Can Protect Against Log4j Vulnerabilities

    Step One: Find Log4j Wherever it Exists

    You need to investigate every internet-facing application, website and system that you own or use. This includes self-hosted installations of vendor products and cloud-based services. Focus on systems that are internet-facing that contain sensitive data. Once you’ve completed assessing your hosted apps and vendor systems, move on to endpoint applications. Java-based apps like WebEx, Minecraft, Citrix and many more.

    Step Two: Patch, Patch, Patch

    Patch in the same order as above: Internet-facing applications, websites and systems. If no patch is yet available for your apps and systems, uninstall.

    The pandemic has forced many endpoints into workers’ home environments, which creates extra challenges for patching those endpoints and keeping them compliant. Operations teams can’t rely on work-from-home folks to patch their own systems, even if they’re given clear instructions. Many of them will just ignore those directions, so you need to adopt systems that enforce continuous compliance.

    Step Three: IT Operations is Essential

    Communicate with your senior leaders. They should be in the know about this one! Senior leadership needs to position their IT operations teams among their most vital employees. Don’t think this is going to go away any time soon—we’re just starting to get a glimpse of what is being tried out there in the wild. Buckle up. 

     

  • U.S. Govt. CX EO | Mozilla Revenue | Log4j Latest

    U.S. Govt. CX EO | Mozilla Revenue | Log4j Latest

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Improving U.S. government CX, how much money Mozilla makes, and the latest on the Log4j/Log4Shell débâcle.

    EO: “Transforming Federal Customer Experience”

    First up this week: The Biden–Harris administration makes an executive order directing its Federal agencies to make 36 specific customer experience (CX) improvement commitments. Much of this involves using technology to improve what DevOps teams would call UX—but government doesn’t like the language of “users.”

    Analysis: Customer-centricity FTW

    Remarkably, a lot of interaction with government still involves paper forms. But this EO shouldn’t merely be about putting existing processes online. Work also needs to be done to solve the underlying complexity and fragmentation of services: Focusing on CX is a helpful frame.

    Brett Samuels: Biden to sign order to streamline government services

    [The] executive order [is] intended to cut back on the bureaucracy around government services for the public such as renewing passports, applying for loans or changing names. [It] affects 36 “customer experience improvement commitments” across 17 federal agencies, [targeting] travel, retirement, business, health and updating personal information.
    …
    For example … to renew their passports online rather than dealing with print forms. [It] will create a single portal for … student loan debt, and small-business owners will have a more streamlined process for … loans, grants and certifications.
    …
    Survivors of natural disasters will also no longer be required to complete forms with several agencies when applying for assistance … and they will be able to file smartphone photos and use virtual inspections when filing claims. [That] could be critical as natural disasters become increasingly devastating, as evidenced by the tornadoes that ravaged Kentucky over the weekend, killing dozens of people and destroying communities.


    Is this a technology problem? Yes, in part—but it’s also a government problem, argues Jennifer Pahlka:

    Surely laws like the 21st Century IDEA Act … which required agencies to … “eliminate or consolidate websites or web pages that are duplicative or overlapping … and ensure that each website or digital service is designed around user needs,” made life easier for the customers of those agencies? What’s that you say? You’re not sure you’ve noticed that? Turns out you’re not alone.
    …
    An EO is one of the many mechanisms our government has at its disposal for what I like to call “magic words.” [But there’s a] gap between magic words and the changes they are supposed to compel. [Failures] happen at such a staggering scale they have become entirely expected.
    …
    We tend to assign the blame … to government technology [but] at some point, you can only improve the customer experience by addressing [underlying] complexity. … Most of the Kafka-esque nightmares people get stuck in happen because each service sees only the burden they impose, and not [what] the customer experiences.


    If, like me, you’re inherently cynical about government efficiency, you’d expect this effort to be a huge failure. But let’s not be too cynical: Warning of “the hazards of selection bias,” here’s ShanghaiBill:

    In the olden days, I took half a day off work to go to DMV and renew my DL. Now I can do it online in 5 minutes. … I pay my property tax online. I renew my business licenses online. I incorporated online.

    $500 Million per Year—But for How Long?

    The Mozilla Foundation has unveiled its revenue forecasts for this year. As in previous years, Mozilla makes the lion’s share of its cash from Google—by making it the default search engine. But there are new revenue sources.

    Analysis: And not a moment too soon

    The future of the browser market should be a strategic focus for DevOps teams. Mozilla’s Googley cash cow might not last. It’s often assumed the maker of Chrome keeps Mozilla alive so it can point to Firefox as “competition.” But with Microsoft’s browser Edging up the usage charts, that might no longer be true.

    Frederic Lardinois: Mozilla expects to generate more than $500M in revenue

    For 2021, the organization forecasts revenue of more than $500 million. What’s maybe most important, though, is that Mozilla’s new products like its Mozilla VPN service, Firefox Relay Premium, Pocket and other commercial initiatives are slowly but surely starting to pay off.
    …
    But for all intents and purposes, Mozilla remains fully dependent on Google for the time being. Diversifying its revenue sources is really the only way for Mozilla to decrease its reliance on a search deal with a company that is both a competitor, thanks to its dominant Chrome browser, and is increasingly out of alignment with Mozilla’s overall philosophy.
    …
    There is a window here for a non-Chrome browser, with users increasingly skeptical about Google’s motivations and Microsoft’s Edge team making a few missteps in recent months. Yet at the same time, Mozilla’s efforts to bring sponsored suggestions and ads to Firefox haven’t necessarily endeared the organization to its own users either.


    Why is Google funding its competitor? u/SerAaron:

    Google is Mozilla’s biggest donor. … It’s basically to avoid any monopoly allegations—so they can point to Firefox and say “look, we have competition”.

    Mozilla takes their money, sure, but … they don’t sell data, and they aren’t beholden to Google.


    But don’t count on that cash continuing to flow, says Maelwryth:

    That money is probably under threat now with Edge going for the market. Why give Mozilla hundreds of millions to show competition when you have actual competition from Microsoft?

    Log4j/Log4Shell Seems Worse Every Time We Look

    The list of popular DevOps tools vulnerable to Log4Shell grows by the hour. Merely finding your exploitable services can be an impossible task—let alone patching them.

    Analysis: Defense in depth is key

    Searching for and patching your vulnerable code is not enough. You’re also going to need to shore up your network defenses, by auditing internet-visible services and by tightening your application firewall rules. And—ironically—keep a close eye on your logs for signs of attempted exploits and IoCs.

    Steven J. Vaughan-Nichols: We Are in So Much Trouble

    Log4j is used a lot. And, when I say a lot, I mean it’s used in hundreds of applications [including] Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark, Struts, Apple iCloud, … LogStash, GrayLog2, … Steam, Twitter, and … VMware.
    …
    Log4Shell is both easy to exploit and can be used to grab complete control of vulnerable servers. … Even as you’re reading this, odds are decent you’re being attacked. [But] even if you patch it in your systems, it’s all too likely to be hiding in other applications.
    …
    Log4Shell may be the worst security problem in a generation. … This is going to be a long, hard ride.


    Hyperbole? Even December 25 isn’t close enough for government work. Catalin Cimpanu:

    The US Cybersecurity and Infrastructure Security Agency has told federal civilian agencies to patch systems affected by the Log4Shell vulnerability by Christmas Eve. … Federal agencies have ten days … to test which of their internal apps and servers utilize the Log4j Java library, check if systems are vulnerable to the Log4Shell exploit, and patch affected servers.
    …
    Despite being a recent vulnerability, it is already considered one of the worst security flaws ever discovered, primarily due to its near-ubiquitous use among enterprise software makers, its simple and easy-to-use exploit, and the ability to hijack systems remotely. … Days after being disclosed, the vulnerability has been massively abused. … The first attacks were seen on December 1.


    There’s always another shoe. Here’s Teri Robinson:

    Maybe Log4j vulnerabilities are like rats—for every one that’s visible, multiple others scurry beneath the surface. … Apache said the fix addressing “CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations,” according to … CVE 2021-45046.

    The Moral of the Story: In the end, the love you take is equal to the love you make.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Bernd Dittrich (via Unsplash)

  • Overcoming Challenges to Automating DevSecOps

    Overcoming Challenges to Automating DevSecOps

    In the last few years, DevSecOps has been widely adopted among organizations looking to get proactive with their security. Traditionally, development teams would continuously implement and deploy new applications into the enterprise and security was an additional bolt-on at the end. 

    However, as the threat posed by cybercrime has increased, organizations have been ‘shifting left’ with their security and weaving it into the beginning of the development life cycle. 

    This offers significantly increased security benefits and means software vulnerabilities can be identified and remediated before they are exposed to the enterprise. The one drawback is around the additional time this takes. As a result, many organizations are also looking to automate their DevSecOps processes. 

    Automation is a critical feature of DevSecOps as it improves efficiency, cuts costs and can streamline operations. However, to get the most out of DevSecOps automation so it delivers real value to the organization, there are several challenges that must be considered first. But what are these challenges and how can they be overcome?

    How to Approach DevOps Security Automation

    When you first move to security automation, you’ll find some security tasks that can easily be automated and others that are harder to fit in. DevSecOps teams will need to use many tools to cover all their bases, but no one wants teams checking lots of different tool outputs. Making it easier to see what’s going on and where risk lies by consolidating security tools and results in a central platform is the way to go. This makes life easier by giving a single pane of glass. Everyone generally agrees that DevOps teams must adopt cybersecurity best practices, but testing takes time. Development teams don’t have time to learn, use and check lots of security tools on top of their existing workload. Automated solutions get to work without any real need for maintenance or management. These solutions trigger tools to work at the right time depending on the outcomes found without any manual input needed. This gives you and your team valuable time back to focus on other important matters.

    Fitting DevSecOps Into Secure CI/CD Pipelines

    The goal of CI/CD pipelines is fast delivery of build and release steps, typically through automation. A key goal of DevSecOps is to alert someone to a new problem as early in that automated process as possible. You need to develop solutions that don’t overwhelm CI/CD pipelines, yet allow flexibility for differing tech stacks, security tools and environments. What you need to ask is, have the latest changes caused any new and important security issues that should be reported before going live?

    Doing DevSecOps Without Constant CI/CD Changes

    Trying to manually run security tools on your entire application source code each day can be time-consuming and hinder your ability to keep up with daily changes. For security checks to keep pace with code delivery in a CI/CD environment, security automation is a must. Security products need to integrate into the development pipeline and enable both the development and security team to work together instead of just throwing things over the fence to each other. 

    Focusing on Issues Raised for Faster Remediation

    All the artificial intelligence and machine learning (AI/ML) in the world isn’t going to remove non-issues better than a manual triage, so here we recommend being resigned to getting stuck. Treat the first run of a set of security tools as a normal security test. Expect hundreds of issues and be prepared to go through them and mark the false positives, duplicates and issues that just aren’t important right now. But do this with the expectation that you’re creating a baseline against which your future pipeline scans can be measured so you’re only alerted to new important issues. By automatically managing and recording the lists of real issues existing in every DevSecOps pipeline run, we can then concentrate on the differences. These new issues need to be flagged and then made easily visible and actionable for all teams.

    Running Multiple Security Tools in CI/CD

    DevSecOps includes setting up many automated security tools to cover the many types of security issues that need to be checked for. Many organizations run multiple security tools. However, as the number of security tools grows and the DevSecOps processes become more complex, many companies then realize the hardest challenge is to make sense of all of the test results within a short period of time. For some companies, this task of bringing together the results from all of these tools is a full-time job, and one ripe for automation. 

    Triaging Security Issues

    False positives and nonsense issues are consistent problems for using security tools. Removing these gives better results and increases the chances that development teams will have a better relationship with DevSecOps processes. Organizations should build a list of issues that don’t need to be actioned and also use automated security tools to flag new issues and differentiate them from the previous backlog of issues. Automating the ‘best fixes’ across all issues also speeds up fix times, while reducing the workload on security teams. 

    Mapping Security to Development Processes

    Dev and DevOps teams are under pressure to get pipelines running smoothly and don’t want security slowing things down or complicating it. By automating security testing in such a way that it can easily handle changes in repos and containers, organizations can overcome this challenge without compromising security. 

    Risk Prioritization

    DevSecOps will increase the number of issues found as well as the speed with which they should be handled. In reality, only a small number of issues will pose a massive risk to the business. Being able to pinpoint the really big issues in real-time helps security know the big risks are being covered and allows developers to focus on the real issues, not lots of small bugs. You can also intelligently determine the changes in risk for quick and automated decisions in the CI/CD pipeline without slowing down the development team.

  • Securing the Software Supply Chain with Behavioral Analysis

    Securing the Software Supply Chain with Behavioral Analysis

    Lately, software supply chains find themselves in a very interesting and uncomfortable position—the industry spotlight—and not in a good way. While significant and costly breaches such as  SolarWinds or Kaseya make front-page news, supply chain attack tactics (e.g. typosquatting or dependency confusion) that target package managers such as npm, PyPI or WinGet can poison downstream applications or compromise developer credentials.  

    The software engineering community has been steadily increasing its use of third-party and open source solutions to help create new offerings. In fact, over the last five years, there has been a 259% increase in the use of open source.

    One reason for this is the simple fact that creating new applications takes time and can strain budgets that are already tight. With open source code, software engineers gain fast access to pre-existing code that’s already demonstrated a proven ability to address the very unique challenge they are looking to solve. By removing the need to reinvent the wheel, the development timeframe is accelerated, the costs are lowered and teams are freed to focus on the creation of new and unique features that the customers demand.

    At this point, turning to third-party code may seem like a no-brainer. However, it can also open the door to potential risks. This is especially true with engineers who, rather than checking software to ensure it is secure and compliant, take the providers’ assurances as gospel. 

    They are not alone. Other groups will try to conduct their initial due diligence at the onset if that’s possible. Soon after they then ease into a “set it and forget it” approach, where they fail to update the code over time. As a result, while the code may appear secure initially, vulnerabilities can emerge over time, and if the software is not continually updated, bad actors can leverage an unprotected window of opportunity. In fact, according to Enterprise Security Group’s “Modern Application Development Security” eBook, 48% of organizations regularly push vulnerable code. The same research also found that 60 percent of organizations have had production application vulnerabilities exploited.

    Organizations conduct static application security testing (SAST) and dynamic application security testing (DAST) to assess pre-production code vulnerabilities. They will also scan with AV before releasing software, examine signing certificates, and code quality. Yet, despite these efforts, they remain challenged by gaps in security processes and the ability to sufficiently inspect large, complex and multi-layered software packages. What’s missing is the ability to conduct in-depth analysis to pinpoint behavioral differences between compiled software versions.  

    Malicious tampering can often involve changing how the software interacts with the system on which it is deployed.  Differential behavior analysis highlighted below makes it easier to detect those unusual changes. For example, having an application suddenly add a pre-install script that downloads files via certutil is highly unusual. If the addition was planned, reasons for the change would be well documented and it would be associated with a significant new feature. For this type of change to happen in any other scenario is suspicious and deserves further investigation before the software is released. In this case, such action would have caught ua-parser-js developer’s account compromise.

    supply chain

    With the software supply chain at risk and with shared code the new normal, what’s needed is software assurance. 

    Software assurance allows teams to inspect every application layer within an installation package for the presence of software tampering, malware and other indicators of software supply chain attacks.   

    Begin the software assurance process with the deep inspection step through recursive package decomposition. The resulting software bill of materials (SBOM) becomes so much more than a build manifest (a list of components that should be in the final build). It becomes the means to determine exactly which components have unacceptable build quality or those that contain unwanted software behaviors. This enables developers to verify the code they are working with and take appropriate action to protect their customers and partners from the majority of software risks. These assessments include:

    • Software grading: The software package and every included component can be graded based on the types and severity of security issues found. Using a grading system makes it easy for software buyers to gauge the level of risk posed by a software package and determine whether additional deployment measures can be taken to mitigate those risks. For developers, component grading makes it easier to prioritize their remediation efforts and aids in the selection of higher-quality third-party components.  
    • Application hardening: Activating specific build or compiler options can mitigate vulnerabilities and protect applications against attack—but only if they are implemented correctly. Assessing whether these mitigations are correctly implemented ensures that these protection mechanisms will work as expected. 
    • Binary behavior analysis: This allows teams to fully understand how each software component interacts with the system on which it will run. The ability to understand third-party component behaviors, even when developers have no access to the source code, delivers a more comprehensive understanding of supply chain risks. Checking for unusual software behaviors or unexpected behavioral changes between one version and the next is often the only way to identify software tampering by a malicious actor. The ability to have this final check before distribution enables software developers to validate that the software being released is trustworthy.

    For companies that are using software to create business value or producing software that is mission-critical for their customers, turning a blind eye or making software assurance assumptions can no longer be tolerated. In today’s climate where increasingly sophisticated threats loom, inspecting software and monitoring updates for signs of tampering, malicious or unwanted additions is essential and must become a part of standard software development practice. 

  • Does Your Organization Need a Data Diet?

    Does Your Organization Need a Data Diet?

    The scenario is all-too-familiar: There’s a security breach, and afterward, the affected organization asks what it must do to better protect its data.

    But what if that organization never collected and stored that sensitive information in the first place? Often, the best defense against an embarrassing and costly breach is to collect only data that is essential to an entity’s mission.

    Some who work in computer privacy circles refer to this as “going on a data diet.” They know the temptation is great for organizations to bulk up on data of all kinds. After all, storage costs are low. With the move to the cloud, an organization doesn’t even need to invest in hardware and upkeep to store the information it collects. So why not grab whatever data a customer or client is willing to provide?

    Because we live in a time when the only sensible approach to computer security is to wonder when your entity might be breached—not if. That’s why it pays to not only go on a data diet, but to adopt a regimen for keeping your organization’s databases lean and, as a result, your customer relationships healthy. 

    Too often, companies collect data just because there’s a chance it might be useful or they think it’s harmless. I’ve seen this play out firsthand. Sales departments seek any small edge they can find; who knows what small shard of data might prove useful in the pursuit of potential customers? The same is often true of an organization’s marketing unit, whose people are working hard to reach a broader audience for their product or message. If information flows freely from the potential and current customers, they collect it in pursuit of any leg up in the competition for people’s attention.

    But there are costs to an organization beyond the price tag of data storage. That includes reputational risk and exposure to lawsuits and the distractions these and other problems bring. More info might seem better—until hackers have broken into a system and the head of an organization is forced to explain why that data was gathered in the first place.

    Weaponizing Seemingly Harmless Data

    It was not that long ago that there seemed to be two distinct types of data: In one bucket there was personally identifiable information (PII) such as a name, address, phone number and email address and critical financial information like bank account and credit card numbers. Throw in personal health information (PHI) and other obviously sensitive information; everyone knew this sort of data must be zealously safeguarded.

    The second bucket was everything else: The seemingly extraneous bits we all leave behind living in a digital world such as what time you visited a website, the language you choose for a website, what products you viewed. Think of it as a mop bucket—messy if spilled, but not vital to corporate security.

    But those distinctions have blurred in recent years. An individual piece of data might seem innocuous, but not when combined with other sets of data that do not fall under the definition of PII as set out by law. What makes Big Data so promising to an organization’s sales and marketing teams is also what makes it potentially dangerous to collect: The connections that a clever algorithm and today’s awesome computing power can deliver crawling through vast data stores. If you link information that is personally sensitive (a person’s tastes or habits or activities) but not technically PII, it can be weaponized to harm individuals.

    As an example, think of the tracking data offered by people’s cell phones. If a person leaves the same street address at around 8:30 every morning and makes the same drive back and forth, it’s quickly obvious where someone lives and where they work. Given other open records, a hacker with dark motives can find our commuter’s name and other information. It’s not a big jump to connect this information to crimes like theft or even stalking and blackmail.

    Adopting a New Data Discipline

    Given the risks, forward-looking organizations are now adopting privacy programs that force their teams to ask the hard questions before collecting information. For these privacy practitioners, any time someone internally proposes that a new piece of data be grabbed or used in a new way, a conversation ensues about its usefulness and the validity of collecting it.

    Maybe the policy is conducted through an internal privacy committee that meets to debate its appropriateness. Maybe it’s a rigorous process that means checks and balances throughout an organization. But whatever the method, only data that can be justified as fundamental to an organization’s operation is collected.

    Another idea gaining popularity among those focused on privacy concerns: A process for jettisoning old data. (An example: Does an organization need to know a person’s last several addresses or just the current one where they can be reached?)  Under such a data diet, all information that an organization continues to store beyond its immediate use requires a specific justification.

    After all, the information you don’t have is the easiest to protect.

  • DevOps Teams Struggling to Keep Secrets

    DevOps Teams Struggling to Keep Secrets

    A growing number of organizations are suffering security incidents related to exposed secrets in DevOps CI/CD pipelines, according to a recent ThycoticCentrify report.

    The study paints a troubling picture: Only 5% of survey respondents said most of their development teams use the same secrets management processes and tools.

    The incidents run the gamut, from secrets published in the clear in public cloud code repositories to insecure third-party code to vulnerabilities in the organization’s own code or configurations.

    The problem is often made worse by security teams not knowing what to protect because development teams have their own cloud infrastructure or the ability to create internet-facing applications directly without security teams’ knowledge.

    “The reason why we see so many security incidents related to exposed secrets in DevOps is that using hardcoded passwords and keys is the easy path to getting things to work,” said Joseph Carson, chief security scientist and Advisory CISO at ThycoticCentrify. “We need to make it just as easy for DevOps to use secrets vaults so even when secrets get compromised they are no longer usable, valid or expired.”

    Carson said moving away from persistent privileges to just-in-time, on-demand privileges will reduce these types of misconfigurations and attacks.

    “Unfortunately, many developers continue to use hardcoded passwords or static keys that, when compromised, allow an attacker to gain control of the application resulting in either data exposure, ability to change the application configuration or move laterally around the network,” he added.

    John Bambenek, principal threat hunter at Netenrich, said the fundamental problem is that when organizations go to agile and DevOps, they remove all friction to moving code quickly—and that almost always includes removing security.

    “That means security is inherently reactive instead of being embedded into engineering, and that’s assuming security even knows what to protect in the first place,” he said. “It’s the natural consequence of removing security from the equation and expecting that it’ll just work on the backend. That approach hasn’t worked for decades; a JIRA board isn’t going to suddenly fix that problem.”

    From Carson’s perspective, secrets management is the ability to move away from hardcoded passwords or static keys to just-in-time privileges or one-time-use passwords so even when comprised they cannot be used.

    “Many privileged access management solutions that protected privileged access for years have extended functionality to developers to help move the value into DevOps so they can manage credentials for applications, databases, CI/CD tools and services without causing friction in the development process,” he said.

    Approaches like privileged access security helps enable API-as-a-service and provides instant availability of secrets, SSH keys, certificates, API keys and tokens.

    Secrets Management

    Bambenek added the problem isn’t choosing a secrets management process or tool, but rather that they aren’t in place at all.

    “Pick something that will keep keys and secrets out of public cloud repositories that developers will use that allows for quick and easy rotation of keys as the need arises,” he said. “The most important thing is that it’s the same tool everywhere that operates in a near-invisible process for developers. Four decades of software vulnerabilities have taught me that if you rely on the developers to do the secure thing, you’ll consistently be disappointed.”

    One Wrong Step

    Davis McCarthy, principal security researcher at Valtix, pointed out developers are not security practitioners by trade.
    “Developers are building new ideas, and no good idea comes without experimentation,” he said. “Experimenting with a new technology stack may lead to misconfigurations that unintentionally expose data.”

    McCarthy said shared SSH keys between users create a larger attack surface, and users will store secrets differently creating multiple attack vectors.

    If one user’s endpoint becomes compromised and their SSH keys are exfiltrated, it’s difficult to know what the first mitigation steps are. Between insider threats and malware, the wrong step could lead to a breach.

    “Whereas a software engineer may hold great knowledge of secure coding methodologies, web application attacks and encryption, they are still focused on executing their craft,” he said. “Organizations that do not build security into the development life cycle create a systemic risk that leads to failures in best practices.”

    McCarthy added organizations can centralize and embed secrets management into the tools already in use in CI/CD pipelines by creating an inventory, auditing it and frequently rotating secrets to sensitive environments.

    Bambenek said the short answer is to make sure the infrastructure of CI/CD pipelines is centrally managed and controlled without the ability of engineering teams to ‘roll their own.’

    “No security control will remain intact if someone can pick their own tools without any real governance,” he said. “CI/CD processes don’t require engineering teams to manage the backend of the deployment pipeline which means someone can ensure those functions are in place to begin with.”