Category: IT as Code

  • Bridging the AppSec and DevOps Disconnect

    Bridging the AppSec and DevOps Disconnect

    Research estimates that cybercrime is going to cost the world $10.5 trillion annually by 2025, so it is no surprise that cybersecurity has become a top priority for business leaders. 

    Today, security teams are striving to harden their systems against cyberattacks and improve their resilience through more employee security training, incident response programs and the implementation of technology to monitor and detect threats. 

    Many organizations are also adopting a shift left security strategy, where security is being addressed earlier in the application development life cycle. These organizations are realizing that finding security issues in their software after it’s released is much riskier and more costly. After all, the longer a bug is in the wild, the longer an attacker has to find and exploit it. By implementing a shift left security strategy, the window of opportunity for an attacker decreases or can be completely removed if a security issue is addressed in the development stages, while also offering organizations significant security benefits and cost savings. 

    Shifting left is a security testing strategy where DevOps and security teams work in tandem to identify and address vulnerabilities during early development stages. This process significantly improves security as it means vulnerabilities are identified and mitigated before they are introduced to the enterprise environment.

    However, one of the biggest challenges organizations face when adopting a shift left security strategy (DevSecOps) is encouraging security teams and DevOps teams to work together.

    Encouraging AppSec and DevOps Collaboration

    Until recently, DevOps teams and AppSec teams were completely siloed. DevOps teams worked on accelerating the speed at which applications are released into the enterprise; application security teams only got involved in the later stages. However, this process has caused security concerns, particularly as more organizations move to agile development. 

    When security teams eventually got their hands on new applications and updates, security flaws were inevitably found that caused delays and friction with DevOps teams. 

    However, as organizations begin to embrace DevSecOps and shift left strategies, these two previously siloed teams are being forced to work together more closely. So, how can the friction be reduced and the disconnect bridged?

    Culturally, some ingrained attitudes and behaviors challenge the success of any DevSecOps efforts. Security teams have seen DevOps processes accelerate the speed at which software is delivered, but without security considerations, while DevOps teams experienced security slowing down processes and giving inconsistent results and feedback on security issues. Each party has their own manager to please; their own set of metrics that they’re measured against and a priority list as long as their arms already. Both teams follow different processes and, crucially, use different tools. DevOps can’t get around the security tool complexity and lack of integration with their existing toolset and security teams have no control over the CI pipeline to best implement security assurance.

    One of the best ways to overcome this friction is through better technology, process and culture that enables collaboration between teams. First, DevOps teams do care about security, but it might be lower on their priority list. Security teams must understand that DevOps teams care about code, quality and efficiency. They are not security experts and just because an organization is shifting left doesn’t mean they need to become one. Instead, organizations should implement tools for DevOps where security checks can be easily embedded.

    So, what can organizations do to improve the collaboration between AppSec and DevOps teams to achieve DevSecOps?

    DevSecOps is a culture and it requires a shift in thinking. Security teams and DevOps teams need to understand each other’s priorities and understand that by working together they are improving the overall security of the organization and decreasing the risk of cyberattacks. Does the DevOps team understand the potential risks a vulnerability could have on the organization overall? If not, security teams need to educate them.  

    Security automation tools offer great support when organizations are moving to DevSecOps, as they allow bugs and vulnerabilities to be flagged automatically as applications are being developed. This will allow them to be addressed before services go live; however, because they generate frequent alerts, this could add to DevOps’ already busy workloads. By deploying tools which can be synced with CI/CD pipelines and provide custom advisories so DevOps teams can address bugs themselves without having to ask AppSec teams, this can significantly speed up the process, reduce friction and ease workloads.

  • Log4j: It’s All About the Supply Chain, Baby!

    Log4j: It’s All About the Supply Chain, Baby!

    In 2021, the security story in DevOps and DevSecOps has been the supply chain. So, it’s only fitting that we are currently experiencing the mother of all supply chain issues with the Log4j Log4Shell RCE vulnerability to close out the year.

    I won’t waste your time rehashing what Log4j is, why it’s so dangerous and warning you to test all of your own and third-party apps to make sure you’re insulated against this. I will say, though, that I don’t think Log4j is a bug—I think it is a defect in design. Worse, as we are seeing, there are multiple defects in design. All of you Java haters out there—back in the box. Java is no worse or better than any other language that we regularly use to build applications.

    So, what exactly is so special about supply chain security? Why do DevSecOps teams, developers and our security incident management teams have to interpret this as a warning and stay at the top of their game from now on? It’s simple. In the age of SaaSOps; in a world where we Frankenstein applications by cobbling and stitching together numerous third-party components (some open source, some not), we are increasingly reliant on code, even in our own applications, that we did not write. We probably didn’t test it, either. We rely more than ever on the “community” and on unknown third-party suppliers. It has always been a leap of faith with these apps and components, and this year some of those chickens have come home to roost.

    The White House executive order on improving the nation’s cybersecurity calls for a software bill of materials (SBOM) for applications; that is, as they say at the tables in Las Vegas, a fine start, but we need more. If we really had a BOM included with the apps we use (and that includes SaaS apps), how many of you are actually going to read it and act on that information? And besides including all these components in a BOM, what further obligations do suppliers have to ensure that they are replacing and updating vulnerable code?

    This issue becomes increasingly acute with open source. My partner and colleague Mitchell Ashley recently asked if too much open source was a bad thing. I don’t know if I would go that far; the more the better, as far as I am concerned. But we do need better monitoring and management of the open source code that’s out there. More testing before it goes GA, maybe? Yeah, the Cadillac open source projects might have the money and time to do that, maybe, but take Log4j as a perfect example: It’s two guys maintaining it in their free time and trying to do right by the community. How can they possibly keep up?

    I used to think that the answer was in the repos. My friends at Sonatype had a component firewall that would not let you download an old or vulnerable version of code without warning. That was fine if you were downloading it for the first time. But if it was already embedded—then what? Can we scan for older components in an app? My friends at JFrog can do something similar and more with Xray. And Sonatype’s Nexus platform also allows current and prospective users to scan older apps for vulnerabilities—including scanning for Log4j, absolutely free. But is that enough?

    All of these questions involve open source components. What about third-party SaaS apps? Often, we don’t even know what components or code are in those SaaS apps. Again, it is a leap of faith we are making when using them. 

    These supply chain security issues are a problem that we are just beginning to grasp. And while 2021 has been the year the stuff hit the fan and Log4j may be remembered as the biggest of the first, I, for one, am afraid to think about what else is coming. 

    I do know that the only way to tackle this and any future supply chain issues is if developers, DevSecOps, DevOps, SREs and our security incident response teams all work together. Just like most security issues, supply chain security is all of our responsibility.  I hope Log4j hasn’t bitten you too hard, and that you are on top of your exposure. But let’s start thinking about the next one and how we can better prepare for it.

  • Log4j: Is There Such a Thing as ‘Too Much’ Open Source?

    Log4j: Is There Such a Thing as ‘Too Much’ Open Source?

    The Log4j vulnerability got me thinking: Is there such a thing as too much open source?

    Before anyone immediately fires off a flaming email, rage tweet or scathing blog post, hear me out for a moment. If you know me, you know that I am an open source fanatic. I’ve been asked many times, “Should we use open source software and, if so, how much?” My response was (and still is) that open source software is a massive source of innovation and I can promise you, whether or not you use it, know that your competitors certainly are!

    I could go on and on extolling the virtues of open source and how it enabled the cloud, reshaped software architectures, created new product categories and gave us valuable tools like telemetry and increased visibility into cloud-native apps, infrastructure and beyond.

    When I first heard about the Log4j vulnerability on December 9, like many others, I quickly informed the tech leaders, engineers and SREs in my network about the serious potential risks such a zero-day vulnerability in the widely-used Log4j tool represented. In one way—probably the only way—the Log4j exploit shared a very important characteristic with the SolarWinds supply chain attack of late 2020: The widespread distribution and ubiquitous use of the software across applications, systems and organizations. Arguably, Log4j is used much more widely.

    This similarity raised questions I hadn’t fully considered in the context of such widely used software. Can we use too much open source software? Is Log4j too prevalent in its use and, therefore, a single point of failure? How bad would the Log4j situation be if a fix wasn’t immediately available or if one wasn’t forthcoming?

    The answer I kept coming back to is a resounding ‘no’. I still don’t think there’s such a thing as using too much (insert any open source project name) software. Log4j—and so many other open source software projects—are massively beneficial to us, and we virtually couldn’t function without these systems, tools and solutions in today’s age of ubiquitous software.

    What the current Log4j vulnerability, recent software supply chain attacks and other cybersecurity incidents with widespread impact highlight is our need to quickly respond to security incidents and take remediation steps anywhere they are found in the software stack. And that includes open source software. High-impact security vulnerabilities can occur anywhere across the numerous software stacks and services we use, though they have even greater impact when they occur in widely-used infrastructure, reusable components and microservices.

    In a world of infrastructure-as-code (IaC), it’s no longer a matter of patching a particular set of routers or servers. Unfortunately, much of the software stack is not containerized or built using smaller, easily replaceable containers and microservices—at least not yet. Our mindset and processes going forward from this incident must change, and we have to be prepared to respond to and remediate vulnerabilities both horizontally and vertically across the software stack. In certain circumstances, we have to do this on a large scale.

    As is common with active open source projects, Log4j contributors responded quickly with a readily available fix. The real answer to my original question above is that few are as prepared to respond to an important security vulnerability as those incredible, hard-working, active and engaged open source software developers.

  • Log4j Puts Effective IT Operations at Center Stage

    Log4j Puts Effective IT Operations at Center Stage

    News of the Apache Log4j vulnerability exploit is striking fear into the hearts of both software makers and users. Log4j is the most popular Java logging service used today, with over 400,000 GitHub downloads and has been embedded in most internet services and products from companies all over the world, including Apple, Amazon, Cloudflare, Steam, Tesla, Twitter and Baidu, just to name a few.

    Exploiting this vulnerability is simple and allows threat actors to control Java-based web servers and launch remote code execution (RCE) attacks. New variations of the original exploit are being introduced rapidly—over 60 in less than 24 hours. The attack surface grows by the minute. Good credential management and advanced authentication will not help as this exploit is pre-authentication, which means an attacker doesn’t need to authenticate to your web applications to attack. Your web server is vulnerable.

    The good news is your organization likely has a team of seasoned IT operations pros that can help protect against the worst of Log4j’s devastation.

    How IT Operations Can Protect Against Log4j Vulnerabilities

    Step One: Find Log4j Wherever it Exists

    You need to investigate every internet-facing application, website and system that you own or use. This includes self-hosted installations of vendor products and cloud-based services. Focus on systems that are internet-facing that contain sensitive data. Once you’ve completed assessing your hosted apps and vendor systems, move on to endpoint applications. Java-based apps like WebEx, Minecraft, Citrix and many more.

    Step Two: Patch, Patch, Patch

    Patch in the same order as above: Internet-facing applications, websites and systems. If no patch is yet available for your apps and systems, uninstall.

    The pandemic has forced many endpoints into workers’ home environments, which creates extra challenges for patching those endpoints and keeping them compliant. Operations teams can’t rely on work-from-home folks to patch their own systems, even if they’re given clear instructions. Many of them will just ignore those directions, so you need to adopt systems that enforce continuous compliance.

    Step Three: IT Operations is Essential

    Communicate with your senior leaders. They should be in the know about this one! Senior leadership needs to position their IT operations teams among their most vital employees. Don’t think this is going to go away any time soon—we’re just starting to get a glimpse of what is being tried out there in the wild. Buckle up. 

     

  • U.S. Govt. CX EO | Mozilla Revenue | Log4j Latest

    U.S. Govt. CX EO | Mozilla Revenue | Log4j Latest

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Improving U.S. government CX, how much money Mozilla makes, and the latest on the Log4j/Log4Shell débâcle.

    EO: “Transforming Federal Customer Experience”

    First up this week: The Biden–Harris administration makes an executive order directing its Federal agencies to make 36 specific customer experience (CX) improvement commitments. Much of this involves using technology to improve what DevOps teams would call UX—but government doesn’t like the language of “users.”

    Analysis: Customer-centricity FTW

    Remarkably, a lot of interaction with government still involves paper forms. But this EO shouldn’t merely be about putting existing processes online. Work also needs to be done to solve the underlying complexity and fragmentation of services: Focusing on CX is a helpful frame.

    Brett Samuels: Biden to sign order to streamline government services

    [The] executive order [is] intended to cut back on the bureaucracy around government services for the public such as renewing passports, applying for loans or changing names. [It] affects 36 “customer experience improvement commitments” across 17 federal agencies, [targeting] travel, retirement, business, health and updating personal information.
    …
    For example … to renew their passports online rather than dealing with print forms. [It] will create a single portal for … student loan debt, and small-business owners will have a more streamlined process for … loans, grants and certifications.
    …
    Survivors of natural disasters will also no longer be required to complete forms with several agencies when applying for assistance … and they will be able to file smartphone photos and use virtual inspections when filing claims. [That] could be critical as natural disasters become increasingly devastating, as evidenced by the tornadoes that ravaged Kentucky over the weekend, killing dozens of people and destroying communities.


    Is this a technology problem? Yes, in part—but it’s also a government problem, argues Jennifer Pahlka:

    Surely laws like the 21st Century IDEA Act … which required agencies to … “eliminate or consolidate websites or web pages that are duplicative or overlapping … and ensure that each website or digital service is designed around user needs,” made life easier for the customers of those agencies? What’s that you say? You’re not sure you’ve noticed that? Turns out you’re not alone.
    …
    An EO is one of the many mechanisms our government has at its disposal for what I like to call “magic words.” [But there’s a] gap between magic words and the changes they are supposed to compel. [Failures] happen at such a staggering scale they have become entirely expected.
    …
    We tend to assign the blame … to government technology [but] at some point, you can only improve the customer experience by addressing [underlying] complexity. … Most of the Kafka-esque nightmares people get stuck in happen because each service sees only the burden they impose, and not [what] the customer experiences.


    If, like me, you’re inherently cynical about government efficiency, you’d expect this effort to be a huge failure. But let’s not be too cynical: Warning of “the hazards of selection bias,” here’s ShanghaiBill:

    In the olden days, I took half a day off work to go to DMV and renew my DL. Now I can do it online in 5 minutes. … I pay my property tax online. I renew my business licenses online. I incorporated online.


    $500 Million per Year—But for How Long?

    The Mozilla Foundation has unveiled its revenue forecasts for this year. As in previous years, Mozilla makes the lion’s share of its cash from Google—by making it the default search engine. But there are new revenue sources.

    Analysis: And not a moment too soon

    The future of the browser market should be a strategic focus for DevOps teams. Mozilla’s Googley cash cow might not last. It’s often assumed the maker of Chrome keeps Mozilla alive so it can point to Firefox as “competition.” But with Microsoft’s browser Edging up the usage charts, that might no longer be true.

    Frederic Lardinois: Mozilla expects to generate more than $500M in revenue

    For 2021, the organization forecasts revenue of more than $500 million. What’s maybe most important, though, is that Mozilla’s new products like its Mozilla VPN service, Firefox Relay Premium, Pocket and other commercial initiatives are slowly but surely starting to pay off.
    …
    But for all intents and purposes, Mozilla remains fully dependent on Google for the time being. Diversifying its revenue sources is really the only way for Mozilla to decrease its reliance on a search deal with a company that is both a competitor, thanks to its dominant Chrome browser, and is increasingly out of alignment with Mozilla’s overall philosophy.
    …
    There is a window here for a non-Chrome browser, with users increasingly skeptical about Google’s motivations and Microsoft’s Edge team making a few missteps in recent months. Yet at the same time, Mozilla’s efforts to bring sponsored suggestions and ads to Firefox haven’t necessarily endeared the organization to its own users either.


    Why is Google funding its competitor? u/SerAaron:

    Google is Mozilla’s biggest donor. … It’s basically to avoid any monopoly allegations—so they can point to Firefox and say “look, we have competition”.

    Mozilla takes their money, sure, but … they don’t sell data, and they aren’t beholden to Google.


    But don’t count on that cash continuing to flow, says Maelwryth:

    That money is probably under threat now with Edge going for the market. Why give Mozilla hundreds of millions to show competition when you have actual competition from Microsoft?


    Log4j/Log4Shell Seems Worse Every Time We Look

    The list of popular DevOps tools vulnerable to Log4Shell grows by the hour. Merely finding your exploitable services can be an impossible task—let alone patching them.

    Analysis: Defense in depth is key

    Searching for and patching your vulnerable code is not enough. You’re also going to need to shore up your network defenses, by auditing internet-visible services and by tightening your application firewall rules. And—ironically—keep a close eye on your logs for signs of attempted exploits and IoCs.

    Steven J. Vaughan-Nichols: We Are in So Much Trouble

    Log4j is used a lot. And, when I say a lot, I mean it’s used in hundreds of applications [including] Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark, Struts, Apple iCloud, … LogStash, GrayLog2, … Steam, Twitter, and … VMware.
    …
    Log4Shell is both easy to exploit and can be used to grab complete control of vulnerable servers. … Even as you’re reading this, odds are decent you’re being attacked. [But] even if you patch it in your systems, it’s all too likely to be hiding in other applications.
    …
    Log4Shell may be the worst security problem in a generation. … This is going to be a long, hard ride.


    Hyperbole? Even December 25 isn’t close enough for government work. Catalin Cimpanu:

    The US Cybersecurity and Infrastructure Security Agency has told federal civilian agencies to patch systems affected by the Log4Shell vulnerability by Christmas Eve. … Federal agencies have ten days … to test which of their internal apps and servers utilize the Log4j Java library, check if systems are vulnerable to the Log4Shell exploit, and patch affected servers.
    …
    Despite being a recent vulnerability, it is already considered one of the worst security flaws ever discovered, primarily due to its near-ubiquitous use among enterprise software makers, its simple and easy-to-use exploit, and the ability to hijack systems remotely. … Days after being disclosed, the vulnerability has been massively abused. … The first attacks were seen on December 1.


    There’s always another shoe. Here’s Teri Robinson:

    Maybe Log4j vulnerabilities are like rats—for every one that’s visible, multiple others scurry beneath the surface. … Apache said the fix addressing “CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations,” according to … CVE 2021-45046.


    The Moral of the Story: In the end, the love you take is equal to the love you make.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Bernd Dittrich (via Unsplash)

  • Overcoming Challenges to Automating DevSecOps

    Overcoming Challenges to Automating DevSecOps

    In the last few years, DevSecOps has been widely adopted among organizations looking to get proactive with their security. Traditionally, development teams would continuously implement and deploy new applications into the enterprise and security was an additional bolt-on at the end. 

    However, as the threat posed by cybercrime has increased, organizations have been ‘shifting left’ with their security and weaving it into the beginning of the development life cycle. 

    This offers significantly increased security benefits and means software vulnerabilities can be identified and remediated before they are exposed to the enterprise. The one drawback is around the additional time this takes. As a result, many organizations are also looking to automate their DevSecOps processes. 

    Automation is a critical feature of DevSecOps as it improves efficiency, cuts costs and can streamline operations. However, to get the most out of DevSecOps automation so it delivers real value to the organization, there are several challenges that must be considered first. But what are these challenges and how can they be overcome?

    How to Approach DevOps Security Automation

    When you first move to security automation, you’ll find some security tasks that can easily be automated and others that are harder to fit in. DevSecOps teams will need to use many tools to cover all their bases, but no one wants teams checking lots of different tool outputs. Making it easier to see what’s going on and where risk lies by consolidating security tools and results in a central platform is the way to go. This makes life easier by giving a single pane of glass. Everyone generally agrees that DevOps teams must adopt cybersecurity best practices, but testing takes time. Development teams don’t have time to learn, use and check lots of security tools on top of their existing workload. Automated solutions get to work without any real need for maintenance or management. These solutions trigger tools to work at the right time depending on the outcomes found without any manual input needed. This gives you and your team valuable time back to focus on other important matters.

    Fitting DevSecOps Into Secure CI/CD Pipelines

    The goal of CI/CD pipelines is fast delivery of build and release steps, typically through automation. A key goal of DevSecOps is to alert someone to a new problem as early in that automated process as possible. You need to develop solutions that don’t overwhelm CI/CD pipelines, yet allow flexibility for differing tech stacks, security tools and environments. What you need to ask is, have the latest changes caused any new and important security issues that should be reported before going live?

    Doing DevSecOps Without Constant CI/CD Changes

    Trying to manually run security tools on your entire application source code each day can be time-consuming and hinder your ability to keep up with daily changes. For security checks to keep pace with code delivery in a CI/CD environment, security automation is a must. Security products need to integrate into the development pipeline and enable both the development and security team to work together instead of just throwing things over the fence to each other. 

    Focusing on Issues Raised for Faster Remediation

    All the artificial intelligence and machine learning (AI/ML) in the world isn’t going to remove non-issues better than a manual triage, so here we recommend being resigned to getting stuck. Treat the first run of a set of security tools as a normal security test. Expect hundreds of issues and be prepared to go through them and mark the false positives, duplicates and issues that just aren’t important right now. But do this with the expectation that you’re creating a baseline against which your future pipeline scans can be measured so you’re only alerted to new important issues. By automatically managing and recording the lists of real issues existing in every DevSecOps pipeline run, we can then concentrate on the differences. These new issues need to be flagged and then made easily visible and actionable for all teams.

    Running Multiple Security Tools in CI/CD

    DevSecOps includes setting up many automated security tools to cover the many types of security issues that need to be checked for. Many organizations run multiple security tools. However, as the number of security tools grows and the DevSecOps processes become more complex, many companies then realize the hardest challenge is to make sense of all of the test results within a short period of time. For some companies, this task of bringing together the results from all of these tools is a full-time job, and one ripe for automation. 

    Triaging Security Issues

    False positives and nonsense issues are consistent problems for using security tools. Removing these gives better results and increases the chances that development teams will have a better relationship with DevSecOps processes. Organizations should build a list of issues that don’t need to be actioned and also use automated security tools to flag new issues and differentiate them from the previous backlog of issues. Automating the ‘best fixes’ across all issues also speeds up fix times, while reducing the workload on security teams. 

    Mapping Security to Development Processes

    Dev and DevOps teams are under pressure to get pipelines running smoothly and don’t want security slowing things down or complicating it. By automating security testing in such a way that it can easily handle changes in repos and containers, organizations can overcome this challenge without compromising security. 

    Risk Prioritization

    DevSecOps will increase the number of issues found as well as the speed with which they should be handled. In reality, only a small number of issues will pose a massive risk to the business. Being able to pinpoint the really big issues in real-time helps security know the big risks are being covered and allows developers to focus on the real issues, not lots of small bugs. You can also intelligently determine the changes in risk for quick and automated decisions in the CI/CD pipeline without slowing down the development team.

  • Stacklet Embeds Collaboration in Compliance-as-Code Platform

    Stacklet Embeds Collaboration in Compliance-as-Code Platform

    Stacklet has added collaboration capabilities to its security and compliance platform that automatically groups related notifications, routes them to the right stakeholders and integrates with existing workflows and collaboration tools.

    The Stacklet platform is based on Cloud Custodian, an open source project that provides access to a domain-specific language that enables an IT team to employ YAML files to manage compliance-as-code.

    Stacklet CEO Travis Stanfield said that approach makes it simpler for organizations that have embraced DevSecOps best practices to comply with a wide range of mandates by shifting responsibility for achieving compliance further left toward DevOps teams.

    The latest version of the Stacklet Platform adds communications capabilities that leverage cloud resource configuration and policy metadata to automatically route notifications and escalations. Customizable notification templates add context for teams to collaborate either via email or integrations with Slack, Microsoft Teams and Symphony instant messaging. The Stacklet Platform can also trigger external workflows in tools such as Jira and ServiceNow to track issue resolution and generate reports.

    Ultimately, the goal is to make it simpler for IT teams to prioritize their compliance remediation efforts by keeping everyone involved continuously updated, said Stanfield.

    The number of compliance and security issues that organizations are experiencing in cloud computing environments is especially problematic. Developers often employ infrastructure-as-code (IaC) tools to provision cloud infrastructure. Unfortunately, they typically lack security and compliance expertise, which then results in large numbers of misconfigurations.

    The Stacklet platform is designed to give DevOps teams a way to prevent compliance and security issues using a compliance-as-code framework that spans multiple cloud platforms rather than requiring DevOps teams to employ a different set of compliance tools for each cloud platform on which applications are deployed.

    It’s not clear yet how far left responsibility for compliance will shift. In theory, organizations could avoid penalties if compliance issues are addressed before an application is deployed in a production environment. However, the teams that manage compliance in larger enterprises today are even further removed from DevOps teams than their cybersecurity counterparts. It may be several years before the cultural divide between these teams is bridged. In the meantime, many DevOps teams are taking it upon themselves to manage compliance as code to prevent unexpected issues from arising at the last minute before deployment.

    Overall, the goal is to enable DevOps teams to enforce compliance policies without slowing down the rate at which applications are built and deployed. The hope is that the adoption of compliance-as-code and DevSecOps best practices will address that issue. In fact, as the pace at which applications are deployed continues to accelerate to support digital business transformation initiatives, the number of potential compliance issues steadily increases. More troubling still, the greater the number of compliance issues there are, the more likely it becomes that cybercriminals will find a vulnerability to exploit.

  • AWS Outage Outrage | Rusty Linux | ARM Latest

    AWS Outage Outrage | Rusty Linux | ARM Latest

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Amazon Web Services falls on its face, Linux’s move to Rust takes the next step, and the FTC stabs another fatal wound in the horrible Arm/Nvidia deal.

    AWS us-east-1 Breaks the Internet

    First up this week: Amazon Web Services’ outage on Tuesday sent scads of DevOps teams scrambling. But what was the cause?

    Analysis: DNS DDoS?

    While we still don’t officially know, there is some chatter to speculate on. Could it be that AWS’s DNS infrastructure is vulnerable to DNS reflection attacks?

    Hello. It’s Dave Lee: From angry Adele fans to broken robot vacuums

    It started early on Tuesday morning. Robot vacuums ceased sucking, Wi-Fi cameras stopped watching and eager Tinder daters were left unable to “swipe right.”
    …
    Within Amazon itself … the unthinkable occurred: grounded delivery drivers were unable to load packages and deliver to customers’ doorsteps, just as the peak Christmas season begins to step up. … Drivers at multiple facilities across the country were sent home … some dreading whatever workload may await.
    …
    An outage at Amazon Web Services … had rippled through the online economy, crippling services used by millions of people. … An “impairment of several network devices” in … US-EAST-1 was the “root cause” of the disruption, Amazon said.


    I was wondering what DevOps’ers learned? Enter Max Rozen:

    AWS took approximately an hour to update their status page to indicate anything was happening. … While our infrastructure is hosted across 20 AWS regions, a single point of failure was hosted in us-east-1.
    …
    What we learned: … I learned the saying, “Friends don’t let friends deploy in us-east-1.” As a result, we’ll be moving some things over to us-east-2 to start with. Secondly, we’ll be adding redundancy and the ability to failover to another AWS region.


    Getting inside the business, here are Eugene Kim and Katherine Long, to go over everything:

    Even inside AWS … information on the outage remained sketchy [and] rumors spread among staff. One AWS employee speculated that the outage was caused by an “orchestrated DNS attack,” while another employee downplayed those concerns.
    …
    According to a screenshot of an internal AWS communique … ”firewalls are being overwhelmed.” … A separate internal note said … the network issue “specifically impacted” Amazon’s internal DNS servers.


    Linux Officially Gets Rust Code

    The Rust language has taken the next step toward being part of Linux kernel development. Some “drivers and other non-core kernel programs” written in Rust can now be PR’ed. But there’s still no plan to go further.

    Analysis: Still needs lubrication

    Rust is much better than C for writing memory-safe code, and coding provable implementations. However, bootstrapping the toolchain is still stupidly hard.

    Steven J. Vaughan-Nichols: Linux is getting more Rust in it by the day

    It wasn’t that long ago that the very idea that another language besides C would be used in the Linux kernel would have been laughed at. [But] this has been coming for several years now. … Why? Because it’s much safer than C, especially at handling memory errors.
    …
    The next “patch series [adds] support for Rust as a second language to the Linux kernel.” … While he’s encouraging a slow but steady approach to introducing Rust into Linux, [Linus Torvalds] has also said that using Rust interfaces for drivers and other non-core kernel programs makes sense. [But] no one’s going to rewrite the kernel’s 25 million lines of C in Rust.
    …
    The Rust code proposed for the kernel now relies on the stable Rust compiler rather than the beta compilers. … As Linux kernel and lead Rust on Linux, developer Miguel Ojeda, put it, “By upgrading the compiler, we have been able to take off the list a few unstable features we were using.”


    How is it “safer than C”? ssokolow shares this fascinating insight:

    Rust tries to encourage that through good API design which allows it to be proven at compile time that the invariants will be upheld, thus avoiding the need to choose between safety and performance. My favourite example … is “the typestate pattern” which lets you teach the Rust compiler to prove correct traversal of any state machine.
    …
    A more mundane example would be the iterator APIs: No need for bounds checking if the iterator looks up the length once at the beginning and the APIs prevent you from wandering out of each entry it hands you. … Rust programs tend to run faster than their C and C++ counterparts because the developers feel comfortable introducing and maintaining more aggressive optimizations.


    “And then a miracle occurs,” quips @dave_universetf:

    I’m intrigued [but] I can’t help but feel sorry for the people doing bootstrapping work. C is hard enough to bootstrap, but Rust is a whole new level of pain.
    …
    Rust is literally not source bootstrappable. [The compiler] that works right now is Rust 1.39. So then you have to compile 18 more Rust compilers to get up to the current version.


    Nvidia’s ARM Woes Continue

    As I’ve said before, this deal really shouldn’t go through. Ownership of the ARM ISA and the core designs would be a huge strategic stick to beat Nvidia’s competition with. ARM chips are an increasing fixture in the datacenter—especially those that value “performance per Watt.”

    Analysis: Stick a fork in it

    Nvidia’s attempt to buy Arm from SoftBank is suffering death by a thousand cuts. And the latest wound comes from the FTC’s Higonokami knife.

    Paul Thurrott: FTC Sues to Block Sale of Arm to Nvidia

    The U.S. Federal Trade Commission … sued to block Nvidia’s $40 billion acquisition of … Arm, citing competitive concerns. … Nvidia announced its intention to purchase Arm from Softbank for about $40 billion, over one year ago, in September 2020. But the deal fell under immediate criticism from the major industry players that rely on Arm’s … designs, and the UK government announced … it would “intervene” for national security reasons.
    …
    Now, it may not happen at all. “We will continue to work to demonstrate that this transaction will benefit the industry and promote competition,” Nvidia said.


    Explain like I’m 5? Here’s swillden:

    ARM has a non-trivial and increasing share in the data center. … ARM is much, much bigger than phones … Arm is an important competitor to Intel and AMD. … Arm licensing IP to Qualcom, nVidia, Mediatek, Unisoc, Samsung, Apple, Google, etc., has been great for many areas of technology.
    …
    Arm acting as a common … designer for a bunch of different chip vendors has been extremely good for … progress in CPU/SoC design. … It would be really unfortunate if a lot of the industry were to shift back to x86, eliminating all of the competition on hardware features that we get from the many ARM chipmakers.


    But what of SoftBank? Ian King runs the numbers:

    Wall Street analysts are all but calling the deal dead. … According to Citibank analysts, the chances of the Nvidia purchase closing are now just 5%.
    …
    The company faces the prospect of losing a jackpot of roughly $74 billion. … Other suitors are unlikely to match such an offer. … So SoftBank’s best alternative may well be an initial public offering for the Arm business—but that is likely to value the company at … $19.8 billion—only about 27% the value of the … Nvidia deal.


    The Moral of the Story: It’s no secret that the both of us are running out of time


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk. (It’s so typical of me to talk about myself—I’m sorry.)

    Image: Ləman Şirinzadə (via Unsplash)

  • Securing the Software Supply Chain with Behavioral Analysis

    Securing the Software Supply Chain with Behavioral Analysis

    Lately, software supply chains find themselves in a very interesting and uncomfortable position—the industry spotlight—and not in a good way. While significant and costly breaches such as  SolarWinds or Kaseya make front-page news, supply chain attack tactics (e.g. typosquatting or dependency confusion) that target package managers such as npm, PyPI or WinGet can poison downstream applications or compromise developer credentials.  

    The software engineering community has been steadily increasing its use of third-party and open source solutions to help create new offerings. In fact, over the last five years, there has been a 259% increase in the use of open source.

    One reason for this is the simple fact that creating new applications takes time and can strain budgets that are already tight. With open source code, software engineers gain fast access to pre-existing code that’s already demonstrated a proven ability to address the very unique challenge they are looking to solve. By removing the need to reinvent the wheel, the development timeframe is accelerated, the costs are lowered and teams are freed to focus on the creation of new and unique features that the customers demand.

    At this point, turning to third-party code may seem like a no-brainer. However, it can also open the door to potential risks. This is especially true with engineers who, rather than checking software to ensure it is secure and compliant, take the providers’ assurances as gospel. 

    They are not alone. Other groups will try to conduct their initial due diligence at the onset if that’s possible. Soon after they then ease into a “set it and forget it” approach, where they fail to update the code over time. As a result, while the code may appear secure initially, vulnerabilities can emerge over time, and if the software is not continually updated, bad actors can leverage an unprotected window of opportunity. In fact, according to Enterprise Security Group’s “Modern Application Development Security” eBook, 48% of organizations regularly push vulnerable code. The same research also found that 60 percent of organizations have had production application vulnerabilities exploited.

    Organizations conduct static application security testing (SAST) and dynamic application security testing (DAST) to assess pre-production code vulnerabilities. They will also scan with AV before releasing software, examine signing certificates, and code quality. Yet, despite these efforts, they remain challenged by gaps in security processes and the ability to sufficiently inspect large, complex and multi-layered software packages. What’s missing is the ability to conduct in-depth analysis to pinpoint behavioral differences between compiled software versions.  

    Malicious tampering can often involve changing how the software interacts with the system on which it is deployed.  Differential behavior analysis highlighted below makes it easier to detect those unusual changes. For example, having an application suddenly add a pre-install script that downloads files via certutil is highly unusual. If the addition was planned, reasons for the change would be well documented and it would be associated with a significant new feature. For this type of change to happen in any other scenario is suspicious and deserves further investigation before the software is released. In this case, such action would have caught ua-parser-js developer’s account compromise.

    supply chain

    With the software supply chain at risk and with shared code the new normal, what’s needed is software assurance. 

    Software assurance allows teams to inspect every application layer within an installation package for the presence of software tampering, malware and other indicators of software supply chain attacks.   

    Begin the software assurance process with the deep inspection step through recursive package decomposition. The resulting software bill of materials (SBOM) becomes so much more than a build manifest (a list of components that should be in the final build). It becomes the means to determine exactly which components have unacceptable build quality or those that contain unwanted software behaviors. This enables developers to verify the code they are working with and take appropriate action to protect their customers and partners from the majority of software risks. These assessments include:

    • Software grading: The software package and every included component can be graded based on the types and severity of security issues found. Using a grading system makes it easy for software buyers to gauge the level of risk posed by a software package and determine whether additional deployment measures can be taken to mitigate those risks. For developers, component grading makes it easier to prioritize their remediation efforts and aids in the selection of higher-quality third-party components.  
    • Application hardening: Activating specific build or compiler options can mitigate vulnerabilities and protect applications against attack—but only if they are implemented correctly. Assessing whether these mitigations are correctly implemented ensures that these protection mechanisms will work as expected. 
    • Binary behavior analysis: This allows teams to fully understand how each software component interacts with the system on which it will run. The ability to understand third-party component behaviors, even when developers have no access to the source code, delivers a more comprehensive understanding of supply chain risks. Checking for unusual software behaviors or unexpected behavioral changes between one version and the next is often the only way to identify software tampering by a malicious actor. The ability to have this final check before distribution enables software developers to validate that the software being released is trustworthy.

    For companies that are using software to create business value or producing software that is mission-critical for their customers, turning a blind eye or making software assurance assumptions can no longer be tolerated. In today’s climate where increasingly sophisticated threats loom, inspecting software and monitoring updates for signs of tampering, malicious or unwanted additions is essential and must become a part of standard software development practice. 

  • DevSecOps for Databases: Data Masking, Cloud Backup, WAF and More

    DevSecOps for Databases: Data Masking, Cloud Backup, WAF and More

    DevSecOps is everywhere. More and more organizations realize that the traditional separation between DevOps and security is wasteful and dangerous. As a result, they are rushing to integrate security into all their software development processes. 

    Adding security to containerized applications, cloud resources and other parts of modern DevOps infrastructure is already well understood. But what about databases? Databases were always—and remain—a critical system component and a prime target for attackers. Here, I’ll review the basics of modern database security and show you how to create a continuous security process for databases in a DevSecOps organization.

    What is DevSecOps?

    DevSecOps incorporates security practices into the DevOps methodology. DevSecOps promotes a ‘security-as-code’ approach via the partnership between release engineers and security teams. Similar to DevOps, the DevSecOps approach focuses on creating innovative solutions for complex software development processes. Teams complete this within the framework of an agile approach.  

    DevSecOps aims to close the gap between security and IT teams while ensuring the speedy and safe delivery of code. Organizations replace the siloed development method with a new approach that promotes the shared responsibility of security tasks during the delivery process. 

    In DevSecOps, two supposedly opposite aims—delivery speed and secure code—are united into one effective process. 

    Database Security Basics

    The term “database security” is an umbrella term used to describe all of the tools created to develop and preserve database availability, integrity and confidentiality. A key responsibility for any DBA is to ensure that the information stored in databases is always secure.

    The goal of database security is to protect:

    • Database management systems
    • Applications that access database information
    • Datastores within databases
    • Hardware servers
    • Virtual database servers, such as cloud-based servers
    • Network access points

    It is critical to strike a balance between security and usability. While you must prioritize data security, information needs to be accessible to offer value to the organization. The more access you provide to the information retained in a database, the more vulnerable the information is. However, an overly secure database is neither practical nor functional.   

    A good way to achieve a balance between accessibility and security is to create meaningful security protocols, create tiered levels of information access and invest in a robust database security and activity monitoring tooling. 

    Common Threats and Challenges

    Here are some of the most common types of database attacks:

    • Insider threats—These involve privileged users misusing their access credentials. An insider could be a negligent employee who unwittingly exposes the database to attack, a malicious insider who abuses the database on purpose or an outside infiltrator who steals credentials to breach the database. The risk of a breach increases as more employees hold privileged access credentials.
    • Injection attacks—These involve inserting arbitrary attack strings into the database queries that HTTP headers or web applications serve. These attacks can be SQL or NoSQL. Secure coding practices and regular vulnerability tests should help protect you against injection attacks.
    • Database software vulnerability exploits—Database software providers regularly release security patches for vulnerabilities in their software. However, if you don’t apply the patches quickly, you prolong your exposure and run the risk that hackers will exploit vulnerabilities in your database. 
    • Backup attacks—If you don’t protect your backup data with sufficient controls, you expose your database to an attack on your backups. 

    The following challenges can exacerbate these threats:

    • Large volumes of data—Organizations must adequately manage increasingly large volumes of data using scalable tools and practices.
    • Sprawling infrastructure sprawl—Organizations are increasingly shifting to hybrid and multi-cloud environments, making their networks more complex and security harder to manage. 
    • Evolving regulations—With the global regulatory landscape changing and growing more complex, it is increasingly challenging to maintain compliance.
    • Skill shortage—The demand for cybersecurity personnel far outstrips supply, and this shortage is only likely to increase in the future. 

    4 Techniques for Continuous Database Security With DevSecOps

    A DevSecOps approach calls for continuous database security during all stages of database development and usage. Let’s explore a few techniques for achieving continuous security.

    Data Masking

    Data masking technology lets you create a structurally similar, anonymized version of an organization’s data. You can use this data for user training or software testing. The aim is to safeguard the real data while using a functional substitute for use cases in which the actual data is not needed.  

    In data masking, you keep the format of the data the same. You only change the values of the data. You can alter the data in various ways including character shuffling, encryption and word or character substitution. Whichever method you choose, you have to modify the value in some way that makes reverse engineering or detection impossible. 

    Use data masking when presenting data to users (whether inside or outside the organization) who don’t need to access the data as part of their role. You should apply data masking to all sensitive information before displaying it in public forums. 

    Use Web Application and Database Firewalls

    Protect your database server from security threats using a firewall that stops access to traffic as a default. The only traffic permitted should come from particular web servers or applications that require access to the information. The firewall must also prevent your database from establishing outbound connections unless they are specifically needed. 

    Protecting the database using a firewall is not enough. You should also deploy a web application firewall (WAF). That’s because SQL injection attacks, which target web applications, can be employed to delete or exfiltrate data from the database. A database firewall may not stop this from occurring if attacks are initiated from an application that is a permitted source of traffic. However, a web application firewall likely will stop attacks of this nature because it can analyze traffic at the application layer and identify malicious query patterns.

    Cloud Backup

    Database backup solutions help organizations safeguard their data with backup database copies in the event of user error, physical hardware failure, data corruption or ransomware attacks. Organizations can ensure their information is always accessible by using database backup tools, even if their central database fails or is compromised. 

    Cloud backup services allow you to send a database copy to an off-site, secondary location for safekeeping if there is a disaster or failure. A third-party service provider typically hosts the secondary data storage systems and servers. This provider bills the backup customer based on capacity used or storage space, number of users, data transmission bandwidth, number of times the information is accessed or the number of servers. 

    Using cloud data backup can help you improve data protection without overwhelming IT staff. This labor-saving advantage could be significant and may even offset some of the added costs connected with cloud backup, including data transmission costs. 

    Secure Database User Access

    You must strive to limit the number of individuals with access to the database to the extreme minimum. Administrators should only have minimal privileges—those they require to carry out their job and only when they require access. 

    For small organizations, this might be difficult to achieve because employees have multiple roles, and there may be only one or two database administrators. However, you should at least manage permissions using roles or groups rather than granting access directly.  

    If you have a large organization, you must think about automating access management via access management software. Access management software can give authorized users a provisional password with the privileges they require every time they want access to the database. This software also logs the activities conducted during that time and stops administrators from sharing their passwords. 

    It is critical to lock down any form of password sharing—this is a common practice by administrators and database users, but it makes accountability and effective database security nearly impossible. 

    In addition, you should follow standard account security procedures: 

    • Enforce the use of strong passwords
    • Store, encrypt and salt password hashes
    • Lock accounts after three or four login retries
    • Deactivate accounts when employees leave or switch to different roles

    DevSecOps and Database Security

    In this article, I explained the basics of DevSecOps and database security, covered the primary threats facing database systems in a modern IT environment and provided four ways you can implement continuous security for databases:

    • Data masking—All modern RDBMS systems offer data masking mechanisms that can reduce the impact of compromised accounts.
    • Using WAFs and database firewalls—Firewalls provide excellent runtime protection and visibility for application-layer attacks.
    • Cloud backup—Backing up databases in the cloud is a continuous process that can enhance availability and reduce the impact of ransomware attacks.
    • Secure database user access—Using least-privilege principles can dramatically reduce the threat surface of a database.

    I hope this will be useful as you extend your DevSecOps practices to critical database systems in your environment.