devopsy.kinsta.cloud

Tag: devsecops

  • DevSecOps Will Cross the Chasm in 2022

    DevSecOps Will Cross the Chasm in 2022

    We’ve been talking about DevSecOps and shift-left security for years. Although this approach probably didn’t “cross the chasm” in 2021, we did see some very telling milestones. Cybersecurity VC funding surged to record heights with a focus on DevOps and cloud security and the need for DevSecOps became glaringly evident with the Log4j vulnerability. On a more personal note, our own open source security project Checkov has surpassed three million downloads.

    Much of this momentum was inevitable. Those in infrastructure and reliability circles have already been adopting and touting the benefits of shifting left and automating as much as possible, so in many ways, it’s expected that security would follow suit. Undoubtedly, the pandemic has also accelerated the need for developers to work more autonomously without having security personnel and processes acting as barriers to productivity and velocity.

    We’re confident that 2022 will see the silos between development and security teams continue to crumble as developer-led security practices become the norm for cloud-native organizations.

    DevSecOps finally crossing the chasm doesn’t mean that every enterprise and traditional organization will shift security left and adopt security best practices. It will, however, give them competitive advantages over those who don’t—both in decreased security costs and increased developer productivity and, thus, time-to-market.

    What else is in store for the future of DevSecOps?

    Rise of the DevSecOps Job Title

    Fewer than 5,000 people on LinkedIn currently have “DevSecOps” in their job title, yet there are over 20,000 current openings for DevSecOps roles. In 2022, expect to see more of those positions filled. This means security teams across industries will conduct fewer manual security audits, there will be a considerable influx of homegrown DevSecOps tooling and point solutions will likely begin to consolidate into single platforms. 

    Blurred Lines Between Application and Infrastructure Security

    Until recently, application security was a very well-defined (albeit fractured) space focusing on securing the custom code and open source packages that make up applications. However, with the rapid adoption of cloud-native applications, the lines between application and infrastructure security are blurring. We expect to see this trend continue as more engineers take on more infrastructure-related projects, vendors start catering to use cases outside of their core competencies (through acquisitions and in-house development) and the role of DevSecOps continues to expand within organizations.

    Infrastructure-as-Code: The Great Cloud Migration’s Next Chapter

    We’ve been talking about the great migration to the cloud for years. At this point, many companies are ready to move on to the next chapter: Infrastructure-as-code (IaC). As a result of this trend, DevSecOps will become much more important, as security needs to be baked earlier in development phases or risk being left behind. In addition, security teams will need to become more well-versed in development technologies and practices to provide the proper guidance for the new way applications are built and deployed. 

    More Software Supply Chain Attacks

    Hackers have been targeting retailers and security vendors for years, exploiting a minor weakness to gain access and move laterally into sensitive data. In the past year, however, software supply chain attacks stepped into the spotlight due to multiple supply chain attacks. We expect this trend to continue, which will, in turn, put more focus on securing supply chains. DevSecOps, which has focused mostly on tools and practices for securing the code and infrastructure, will expand to include the supply chain mechanism. 

    If 2021 was the year of hype for DevSecOps, we believe (and hope) that these best practices are embraced en masse across industries. The benefits of shift-left security are well documented: The number of high severity incidents is significantly reduced, the potential attack surface is minimized, compliance efforts are simplified and the time to remediation is lowered. Organizations also save money by catching misconfigurations and vulnerabilities earlier in the software development life cycle while at the same time gaining time back with tools, both open source and commercial, that are empowering developers to move fast and build applications that are more secure and reliable.

  • Best of 2021 – Hiring When You’re Not FAANG

    Best of 2021 – Hiring When You’re Not FAANG

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the twentieth in our series of the Best of 2021.

    Most of us are not FAANG, and we shouldn’t act like we are. Instead of using that, I’m going to use the term we used to use to refer to them: “Web Monsters”, which isn’t 100% accurate, but includes more than just the five FAANG members (Facebook, Amazon, Apple, Netflix and Google) for this article. At some point, we are going to need a new term. Web Monsters doesn’t clearly include Apple or Citi, but FAANG clearly excludes Citi and eBay. But this blog is about something else, so please accept my apologies for that lack of clarity. Assume that ‘Web Monster’ means “Big, popular high-tech company that pays well and looks good on a résumé.”

    There are several angles to “Not Web Monsters” that we can address, but I’m mostly going to focus on DevOps hiring, with a nod to irrelevant sales/marketing techniques. Hiring is an easy target because Web Monsters offer resume-building, pay and benefits that are super attractive in the industry, so they can (right now, at least) reject 99% of applicants and still have a good selection of candidates to choose from. Most organizations cannot compete in even one of those areas, though a few can when it comes to salary and/or benefits.

    We in IT have long had a bad habit of asking our applicants to be more qualified than the average employee. It’s pretty ridiculous at times; things like asking for a candidate to have more years of experience with a technology than the technology itself has actually existed. What happens when you ask for the impossible? Honest, qualified applicants self-select out, and only those who don’t realize the technology hasn’t been around that long and liars end up applying. That’s just bad hiring policy, even if you are a Web Monster. But at smaller organizations where each hire has a larger impact on the organization, it can be devastating.

    It seems worse in DevOps. In fact, all of the hiring items listed here seem worse in DevOps because DevOps is actually a nightmare mish-mash of skill sets. No two people define the job the same because “the job” covers everything from cradle to grave for an application. There is no part of app dev that “DevOps” couldn’t touch. That’s problem number one. Define the responsibilities of the role clearly, don’t ask for everything that it could be. As a rule, an Ansible specialist is not going to be a DevSecOps expert, and you shouldn’t expect them to be. I think, at a minimum, we need to split hiring along my traditional emphasis lines… DevOps and DevOps. If you are writing code and using Checkmarx while GitLab is building the results and you configured it to roll out to test … you’re not a DevOps engineer, even though there’s a bit of ops in there. Your strengths need to be in development and app configuration. Your job description needs to focus on that.

    Could there be roles that need it all? Yes, particularly in the SRE or chaos engineering space; I could see that. But they are certainly not entry-level roles, or even close. Knowledge in the breadth of technologies required to touch the entire DevOps toolchain is developed over years and with a healthy curiosity. And, of course, the person you are hiring should have a similar skill set to those doing the job today. Which is a good sanity check against the “Throw in the kitchen sink” syndrome.

    Next up is take-home tests and shared coding during interviews. We know that you have a vested interest in landing the best people, but if you are doing coding exercises with people who are primarily DevOps, you are wasting everyone’s time. Most DevOps jobs are done with PowerScript and/or bash with YAML files. Other languages used occasionally have no more place in a core job interview than other occasional bits. “It’s about how you think” is a common refrain. I would argue that an assignment that takes an applicant hours to complete (or in several cases I know of, days) is not an efficient way to evaluate the way a prospective employee thinks. And, in at least one case, I’m pretty certain the interview solved an actual problem that employees didn’t—or couldn’t—solve. Ask the candidate questions about how they’d solve problem X. Ask them convoluted questions. They don’t need to get to code level to show you how they think.

    And finally, my nod to bad marketing: Unless you actually are doing the volume of one of the Web Monsters, the fact that Facebook is a customer of vendor X is totally irrelevant. In fact, like the government, most vendors can mark one or more of the Web Monsters as their customers, just because Web Monsters are so big they try a lot of different tools across many different teams. So having them as a customer should be irrelevant, but if it isn’t considered irrelevant, should be expected.

    The team you have today is rocking it. Hire more people like them. You know what works and what you need. Get back to customizing hiring processes to deliver more of what you need and stop over-asking. There are more great people out there looking for roles; don’t scare them off by asking for irrelevant experience. Stick to the core. What are they going to do daily? Require that, then probe for other useful items at interview time. And get more good peeps in the seats!

  • Best of 2021 – From Agile to DevOps to DevSecOps: The Next Evolution

    Best of 2021 – From Agile to DevOps to DevSecOps: The Next Evolution

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the ninth in our series of the Best of 2021.

    As the workforce shifts to remote locations, networks are becoming more diffuse as the edge gets farther away. At the same time, the number of devices and applications on those networks continues to rise. Some estimates suggest that, by 2030, there will be 15 connected devices for every person on the planet.

    The rapid expansion of the remote workforce and their associated application needs has created a significant challenge for both application developers and the IT professionals tasked with deployment and maintenance of applications. Finding the right balance between speed of release and stability and reliability of applications – never an easy task – is paramount for the success of any organization.

    Agile and DevOps evolved to help organizations strike this balance, providing the best possible customer experience in the shortest realistic time frame. But have Agile and DevOps fulfilled that promise?

    The Promise of the Agile Manifesto

    While Agile as a methodology has been known and followed for nearly two decades, the ways in which organizations have applied Agile have been as varied as the organizations themselves.

    The Agile manifesto set out four basic values when it was first written in 2001:

    • Individuals and interactions take precedence over processes and tools.
    • Working software takes precedence over comprehensive documentation.
    • Customer collaboration takes precedence over contract negotiation.
    • Response to change takes precedence over mindlessly following plans.

    Agile processes are applied across a range of development activities, from development of standalone applications to development of cloud-native business applications. The Agile Manifesto also contains several more principles that further explain what the values mean when put in practice.

    These Agile principles can be condensed into a few areas of focus:

    Prioritize the customer over the product

    Agile development requires continuous customer engagement throughout the entire development process, with the end goal being frequent product delivery based on specific customer feedback. While this can present challenges for an Agile team, particularly when customers demand changes late in the development cycle, Agile maintains that the best way to elevate customer experience is to make the customer part of the development process.

    Trust Agile Teams to Self-Organize and Self-Manage

    Agile focuses on smaller teams that remain in constant communication with customers, as well as with each other. These teams should include both developers and those with knowledge of the larger business requirements being met through development. For these teams to work effectively, trust is key. Agile posits that self-organized teams, given leeway to pursue their projects in the way they see fit, result in the shortest development cycles.

    Keep it Simple

    Agile teams require simplicity. This correlates with Agile’s disdain for overuse of process. Agile seeks to minimize the total amount of work required, thus minimizing overall development time.

    Agility Leads to Agility

    Just as Agile teams continuously improve their product and the customer experience, they also constantly improve themselves. Constant evolution of teams and the processes they employ (remember, teams before processes) improves team agility itself.

    While the Agile values and principles seem simple, effective application of the values and principles is another story entirely. Indeed, at times, the implementation of Agile development has violated the very first value of the manifesto by focusing on which tools and processes should be used. But, in an environment where more users are clamoring for more products and services, implementation of Agile methodologies is not just important, it is imperative.

    Building on Agile with DevOps

    The precise meaning of the term itself and the scope of DevOps remain a source of debate. At a high level, DevOps is the marriage of the development cycle with post-development activities, such as deployment and maintenance.

    According to open source contributor Barbara Ericson of Cloud Defense, “Instead of IT operations and software development being siloed off from each other, DevOps breaks down the traditional boundaries that previously existed between them in order to achieve continuous integration and continuous delivery (CI/CD) of quality software features and applications to end users.”

    Just as with Agile, the effective implementation of DevOps requires cultivating a DevOps culture within the organization. However, Agile and DevOps are also not mutually exclusive. That is to say, DevOps is not merely a replacement for or an evolution of Agile. Although Agile and DevOps differ in their structures and approaches, they can work in unison to achieve the ultimate goals of speedy deployment and enhanced customer experience.

    While Agile and DevOps share common goals, they have not always agreed on how to achieve those goals. DevOps differs in many respects from Agile, but, at its best, DevOps applies Agile methodologies, along with lean manufacturing principles, to speed up software deployment.

    One area of particular tension between Agile and DevOps is that the latter relies heavily on tools; in particular, when it comes to the automation of testing and deployment processes. But DevOps can overcome the resistance of Agile developers to tool usage simply by applying Agile principles themselves.

    Effectively, DevOps proponents must convince Agile teams that dogmatic adherence to the underlying principles of Agile is actually inconsistent with Agile in the first place. Ironically, Agile developers who insist that the process is always bad actually violate Agile principles by refusing to acknowledge the benefits offered through change, another Agile principle.

    The challenge is to have the Agile development teams trust in the automation efforts of DevOps, while at the same time encouraging the DevOps team to consider the business goals of deployment rather than pursuing speed of deployment for its own sake.

    With constant communication between the Agile team and DevOps team (another Agile principle), development can achieve a degree of comfort with DevOps tasks and processes. This means that testing and deployment automation can proceed quickly, resulting in a decreased time to market, overall.

    Ultimately, Agile and DevOps can work together most effectively when they recall their shared goals: smaller change increments based on feedback resulting in faster delivery cycles.

    Continued Evolution: DevSecOps

    As networks expand and businesses continue to integrate cloud-based services (in many cases, building multi-cloud environments), security issues become more and more complex. One area in which Agile is lacking is in integrating security into the development process. Unfortunately, in an Agile environment, application security is often an issue addressed after development, rather than as part of the process. Enter the next iteration of DevOps: DevSecOps.

    DevSecOps revolves around three basic principles: ensuring data security while minimizing inconvenience for users in accessing data, using development tools that identify risks as early as possible in the development process and ensuring data encryption using technologies such as secure sockets layer (SSL) and virtual private networks (VPNs).

    While DevSecOps and Agile may also disagree about the priority of tools, security must become an integral part of any development project. This is why Agile, DevOps and DevSecOps must all work together to ensure not only rapid deployment but secure deployment.

    The Next Evolution

    Agile and DevOps were both important evolutions of the software development process, but DevSecOps is the next evolutionary step. Despite a few disparate characteristics, Agile, DevOps and DevSecOps can and should work in unison to achieve their shared goal: the best possible customer experience and short deployment cycles.

  • Only 30% of Orgs Fully Implement DevSecOps

    Only 30% of Orgs Fully Implement DevSecOps

    With the pressure to release more rapidly, security is shifting left within the continuous development pipeline at most organizations. This imperative is increasing with the rise of cyberattacks. Yet both a lack of training and poor visibility are stalling many DevSecOps rollouts, a recent CSA study found.

    DevSecOps is still a relatively new practice and has yet to reach maturity throughout most organizations. Though nearly all companies are on their way, only 30% of security professionals say they fully implement DevSecOps in practice today. That leaves most organizations in some state of DevSecOps ideation and planning.

    CSA recently released its Secure DevOps and Misconfigurations 2021 report which was commissioned by Trend Micro. The report found that misconfigurations commonly arise from default security settings. A lack of internal guidance and accessibility of security resources commonly hinder DevSecOps maturity, too.

    Below, I’ll review the other major takeaways from the report to see how they can inform organizations as they move to implement DevSecOps.

    State of DevSecOps Adoption

    DevSecOps is all about increasing the security around rapid deployment. It literally puts the security into DevOps by shifting security automation left and de-siloing the boundary between DevOps and IT security teams.

    While most engineers realize the benefits of DevSecOps, the state of each organization’s DevSecOps journey is very different. While an impressive 90% of organizations are in some phase of the journey toward DevSecOps, only 30% are implementing DevSecOps while 24% are in the planning phase, 18% are designing and 18% are still refining their DevSecOps strategy. Just over one in ten security professionals say their team has no plans to invest in DevSecOps at all.

    Looking to the future, 42% say they will implement full DevSecOps within the next 12 months. But due to its current nascent state, misconfigurations are still getting through. The primary reason cited for these misconfigurations was flawed or lacking internal guidance (33%).

    Many professionals say there is simply not enough training, support or internal knowledge around vulnerabilities and misconfigurations. Other top reasons for misconfigurations include insecure default settings (18%) and negligence (16%).

    Outside of misconfigurations, groups also report challenges with identity, authorization and access. Ensuring the correct privileges is vital to avoid escalation attacks, which are a prevalent issue. Between 2019 and 2020, 80% of companies experienced a data breach of some kind, many of which were due to misconfigured access controls. OWASP also reported that broken authorization is a top vulnerability for high-traffic web API endpoints.

    Common Threat Mitigation Practices

    To mitigate these threats, development and security teams must maintain best practices, yet certain issues prevent DevSecOps from emerging. For one, 60% of security professionals say their top challenge is insufficient visibility into security or compliance gaps—this is by far the most common challenge—and 11% also cite inconsistent cloud account onboarding. Finally, 10% say time-consuming, slow and/or insufficient architecture holds them back.

    One way to mitigate threats is to introduce routine security reviews more frequently. Yet it’s hard to set a definite benchmark here, as the frequency at which organizations review their cloud infrastructure for vulnerabilities or misconfigurations varies widely. Currently, 22% perform such security reviews daily, 22% monthly, 18% weekly and 23% quarterly, according to the survey. This rate will likely increase as DevSecOps becomes more commonplace and automated within the development life cycle.

    Staying up-to-date with modern security frameworks is another facet impacting DevSecOps adoption; and organizations tend to follow multiple frameworks to inform their security strategy. More than three-quarters (78%) follow the National Institute of Standards and Technology (NIST) guidelines, 67% follow the Center for Internet Security (CIS) benchmarks, 66% follow Cloud Security Alliance (CSA), 54% follow International Organization for Standardization (ISO) and 44% say they follow security recommendations from Amazon Web Services (AWS).

    For example, NIST, the government-operated standards-setting body, recently set cybersecurity guidelines that recommended the use of zero-trust and service mesh across departments. These security guidelines can be thought of as architectural benchmarks, especially for large, highly-regulated industries.

    DevSecOps Training Types

    Finally, investing in community resources and training is another way to increase DevSecOps awareness. Only half of survey respondents reported their DevSecOps best practices resources were moderately accessible—thus, the onus is on leaders to democratize such knowledge within their organization.

    The majority (81%) of respondents cited online articles and training as their primary form of learning about cloud security tools and vendors. Workshops, conferences and webinars follow closely. Organizations also adopt many internal knowledge-sharing methods in response to incidents. A full 85% said they conduct awareness training followed by table-top exercises (52%), attack simulations (45%) and protocol or response framework training (37%).

    Future DevSecOps In The Cloud

    It appears that most teams are still formulating their DevSecOps strategy. But looking to the future, 42% say they will implement full DevSecOps within the next 12 months. Simultaneously, more cloud traction is projected throughout the industry.

    Right now, 40% of organizations have between 41% to 99% of their workloads in the public cloud. And 55% of organizations will have between 41% to 99% of their workloads in the public cloud in the coming year. The type of workloads will also likely evolve in the coming year, as more organizations move to container platforms, Function-as-a-service and other serverless capabilities. To fully reap the benefits these new technologies promise, organizations must undoubtedly respond to a new class of cloud-native vulnerabilities.

    About The Report

    The Cloud Security Alliance (CSA) is a not-for-profit organization with a mission to widely promote best practices for ensuring cybersecurity in cloud computing and IT technologies. The Secure DevOps and Misconfigurations survey was conducted from July 2021 through September 2021 and collected over 900 responses from a global pool of IT security professionals working in various sectors and organizational sizes. For a full copy of the report, you can swap some personal information for a PDF download here.

  • Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users 

    Best of 2021 – Torvalds’ Bug Warning is a Lesson for Linux Users 

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the third in our series of the Best of 2021.

    Linux does, occasionally, raise security concerns. While many users see it as the most secure, robust and versatile operating system available — that’s this writer’s opinion, as well — security precautions still have to be taken.

    A recent, widely publicized case illustrated this point; Linux creator himself, Linus Torvalds, warned against the use of the Linux 5.12 release. He described a “nasty bug,” and wrote that the situation is a “mess,” due to the use of swap files when adding Linux updates. This nasty bug, in fact, had the potential to destroy entire root directories.

    Some of the main takeaways following this “mess” include: tread very carefully when installing early Linux releases, especially those that involve swapping files instead of partitions, and especially, despite Linux’s well-known security advantages, avoid becoming complacent, because Linux security is not always foolproof.

    Hence, while the “state of Linux security today is quite good, and has evolved in a positive way with more visibility and security features built, like many operating systems, you must install, configure and manage it with security in mind; that is how cybercriminals take advantage, [via] the human touch,” said Joseph Carson, chief security scientist and advisory CISO at Thycotic, a provider of privileged access management (PAM) solutions.

    A Patch for Nastiness

    As Torvalds noted a few weeks ago, “most people don’t use a swap file, but a separate swap partition and the bug in question really only happens when you have a regular file system, and put a file on as a swap.”

    “The bad news is that the reason we support swap files in the first place is that they do end up having some flexibility advantages, and so some people do use them for that reason. If so, do not use [release candidate] RC1,” Torvalds wrote. “Thus, the renaming of the tag.”

    After issuing the warning, Torvalds released a patch that he says prevents the bug from destroying swap file systems. However, it may have already been too late for early adopters of release 5.12. Ubuntu, a leading Linux distro, can swap files by default.

    “It is nasty bug if you are still using swap files,” Carson said. “If you do still use swap files, then you could be impacted, resulting in potential data loss or a corrupted system.”

    DevOps teams – or anyone else running Linux and installing patches, whether on multi-servers or on individual workstations – still need, of course, to follow strict best practices. “Like any operating system, security depends entirely on how you use, configure or manage the operating system,” Carson said. “Each new Linux update tries to improve security; however, to get the value, you must enable and configure it correctly.”

    Linux Goodness

    The fact that Torvalds was so forthcoming about the bug, as well as the level of transparency that the Linux kernel offers, also demonstrates one of the many reasons Linux remains popular. Given that the Linux kernel, in one variety or another, is used “not only in about 50% of the internet servers of the world, but also in a substantial part of all our smartphones, it is good to see this level of transparency at ‘root level,” said Dirk Schrader, global vice president, security research at New Net Technologies (NNT), which providers cybersecurity and compliance software.

    “The security of Linux is based on its transparency; the ability to review the code of a distribution,” says Schrader. “Quite often forgotten is that transparency also involves talking about the mistakes, the errors, those nasty bugs.”

    Citing National Institute of Standards and Technology (NIST) vulnerability database statistics, Schrader described how, compared to the Windows family of desktop and server operating systems, for example, the Linux kernel shows better results for overall vulnerabilities. The number of vulnerabilities have also declined over the past four years, while Microsoft’s operating systems do not display the same trend, according to NIST’s national vulnerability database.

    Since Linux’s famous kernel is open source and transparent, it is possible to extrapolate that there are a greater number of potential vulnerability watchdogs compared to those monitoring vulnerabilities in closed systems. Some may argue that Microsoft has been, at times, less successful at detecting vulnerabilities and issuing much-needed patches.

    However, Linux users still must remain vigilant.

    “Still, for any of the Linux distributions, anyone using the early release candidates — RC1 in particular — should make sure that their own development or build process is undergoing change control, so that no mishaps will transfer the nasty bug into a production environment,”  said Schrader.

  • How Log4j Becomes a Serious DevOps Problem

    How Log4j Becomes a Serious DevOps Problem

    The recent discovery of the Apache Log4j vulnerability has wide-ranging implications for anyone who develops software, especially for those in the DevOps realm. What’s most troubling about the vulnerability (CVE-2021-44228) is how prevalent the use of Log4j is. The vulnerability is reported in a vast array of applications and directly impacts numerous Apache projects, including Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark and Struts. There are also numerous Cisco programs impacted by the vulnerability. The list of impacted applications and libraries seems to be almost infinite, extending to Elastic LogStash, GrayLog2, Neo4J, Steam, Twitter and even the VMware Tanzu family of programs. 

    Simply put, any application that uses Log4J for logging and tracing is subject to the identified family of attacks known as Log4Shell. Those attacks prove easy to exploit and can be used to take control of vulnerable servers.

    Why DevOps Practitioners Should be Concerned

    The world of DevOps is all about reiteration, where features and improvements to applications are slipstreamed into production environments. To accelerate development, the QA process is usually limited to the most recent changes.

    In other words, developers, via their pipelines, can often establish a baseline of quality and then speed up their CI/CD pipelines by testing only what has changed since the last iteration of the application. This process ensures an acceptable level of quality without introducing what are perceived to be unnecessary steps.

    However, those processes do not take into account newly discovered flaws in older APIs and libraries, meaning that those flaws could telegraph into deployed applications and leave DevOps teams completely unaware of the impact and possible disruption. Detecting flaws after an application is deployed is like putting the cart in front of the horse.

    Further complicating the situation is that many developers lack a well-defined and well-maintained software bill of materials (SBOM), which means developers may be unaware of the components and/or libraries being used in a deployed application.

    Without an SBOM, it becomes increasingly difficult to remediate an issue with an application—especially if developers are unaware of the issue to begin with.

    Where Should Responsibility Lie?

    More often than not, vulnerabilities introduced into applications become a game of finger-pointing, where developers blame library suppliers or infosec staffers, and infosec blames developers for not performing due diligence on the components they are selecting to incorporate into their applications.

    Of course, finger-pointing only adds to the angst and delays the process of remediation and does nothing to determine where the responsibility for detecting and remediating vulnerabilities lies. DevOps teams need to put these squabbles aside and establish a process that deals with unintentionally introduced vulnerabilities. What’s more, DevOps teams need to invest in DevSecOps to make sure that vulnerabilities are detected and remediated as quickly as possible.

    To avoid the potentially massive disruption that a vulnerability like Log4Shell can cause, those running DevOps should consider:

    • Creating and maintaining an SBOM 
    • Deploying automated tools that scan for discovered vulnerabilities
    • Automatically comparing the SBOM against vulnerabilities
    • Including DevSecOps processes within their CI/CD pipelines
    • Frequently testing applications for flaws
    • Coordinating with cybersecurity personnel when a flaw is discovered
    • Patching frequently

    With more and more vulnerabilities discovered on an almost daily basis, DevOps teams must be ever vigilant about what components are incorporated into their applications. What’s more, with the increased use of low-code/no-code tools and integrated development environments (IDEs), DevOps will also need to consider if those tools use other libraries that can become compromised.

     

     

  • Bridging the AppSec and DevOps Disconnect

    Bridging the AppSec and DevOps Disconnect

    Research estimates that cybercrime is going to cost the world $10.5 trillion annually by 2025, so it is no surprise that cybersecurity has become a top priority for business leaders. 

    Today, security teams are striving to harden their systems against cyberattacks and improve their resilience through more employee security training, incident response programs and the implementation of technology to monitor and detect threats. 

    Many organizations are also adopting a shift left security strategy, where security is being addressed earlier in the application development life cycle. These organizations are realizing that finding security issues in their software after it’s released is much riskier and more costly. After all, the longer a bug is in the wild, the longer an attacker has to find and exploit it. By implementing a shift left security strategy, the window of opportunity for an attacker decreases or can be completely removed if a security issue is addressed in the development stages, while also offering organizations significant security benefits and cost savings. 

    Shifting left is a security testing strategy where DevOps and security teams work in tandem to identify and address vulnerabilities during early development stages. This process significantly improves security as it means vulnerabilities are identified and mitigated before they are introduced to the enterprise environment.

    However, one of the biggest challenges organizations face when adopting a shift left security strategy (DevSecOps) is encouraging security teams and DevOps teams to work together.

    Encouraging AppSec and DevOps Collaboration

    Until recently, DevOps teams and AppSec teams were completely siloed. DevOps teams worked on accelerating the speed at which applications are released into the enterprise; application security teams only got involved in the later stages. However, this process has caused security concerns, particularly as more organizations move to agile development. 

    When security teams eventually got their hands on new applications and updates, security flaws were inevitably found that caused delays and friction with DevOps teams. 

    However, as organizations begin to embrace DevSecOps and shift left strategies, these two previously siloed teams are being forced to work together more closely. So, how can the friction be reduced and the disconnect bridged?

    Culturally, some ingrained attitudes and behaviors challenge the success of any DevSecOps efforts. Security teams have seen DevOps processes accelerate the speed at which software is delivered, but without security considerations, while DevOps teams experienced security slowing down processes and giving inconsistent results and feedback on security issues. Each party has their own manager to please; their own set of metrics that they’re measured against and a priority list as long as their arms already. Both teams follow different processes and, crucially, use different tools. DevOps can’t get around the security tool complexity and lack of integration with their existing toolset and security teams have no control over the CI pipeline to best implement security assurance.

    One of the best ways to overcome this friction is through better technology, process and culture that enables collaboration between teams. First, DevOps teams do care about security, but it might be lower on their priority list. Security teams must understand that DevOps teams care about code, quality and efficiency. They are not security experts and just because an organization is shifting left doesn’t mean they need to become one. Instead, organizations should implement tools for DevOps where security checks can be easily embedded.

    So, what can organizations do to improve the collaboration between AppSec and DevOps teams to achieve DevSecOps?

    DevSecOps is a culture and it requires a shift in thinking. Security teams and DevOps teams need to understand each other’s priorities and understand that by working together they are improving the overall security of the organization and decreasing the risk of cyberattacks. Does the DevOps team understand the potential risks a vulnerability could have on the organization overall? If not, security teams need to educate them.  

    Security automation tools offer great support when organizations are moving to DevSecOps, as they allow bugs and vulnerabilities to be flagged automatically as applications are being developed. This will allow them to be addressed before services go live; however, because they generate frequent alerts, this could add to DevOps’ already busy workloads. By deploying tools which can be synced with CI/CD pipelines and provide custom advisories so DevOps teams can address bugs themselves without having to ask AppSec teams, this can significantly speed up the process, reduce friction and ease workloads.

  • Log4j: It’s All About the Supply Chain, Baby!

    Log4j: It’s All About the Supply Chain, Baby!

    In 2021, the security story in DevOps and DevSecOps has been the supply chain. So, it’s only fitting that we are currently experiencing the mother of all supply chain issues with the Log4j Log4Shell RCE vulnerability to close out the year.

    I won’t waste your time rehashing what Log4j is, why it’s so dangerous and warning you to test all of your own and third-party apps to make sure you’re insulated against this. I will say, though, that I don’t think Log4j is a bug—I think it is a defect in design. Worse, as we are seeing, there are multiple defects in design. All of you Java haters out there—back in the box. Java is no worse or better than any other language that we regularly use to build applications.

    So, what exactly is so special about supply chain security? Why do DevSecOps teams, developers and our security incident management teams have to interpret this as a warning and stay at the top of their game from now on? It’s simple. In the age of SaaSOps; in a world where we Frankenstein applications by cobbling and stitching together numerous third-party components (some open source, some not), we are increasingly reliant on code, even in our own applications, that we did not write. We probably didn’t test it, either. We rely more than ever on the “community” and on unknown third-party suppliers. It has always been a leap of faith with these apps and components, and this year some of those chickens have come home to roost.

    The White House executive order on improving the nation’s cybersecurity calls for a software bill of materials (SBOM) for applications; that is, as they say at the tables in Las Vegas, a fine start, but we need more. If we really had a BOM included with the apps we use (and that includes SaaS apps), how many of you are actually going to read it and act on that information? And besides including all these components in a BOM, what further obligations do suppliers have to ensure that they are replacing and updating vulnerable code?

    This issue becomes increasingly acute with open source. My partner and colleague Mitchell Ashley recently asked if too much open source was a bad thing. I don’t know if I would go that far; the more the better, as far as I am concerned. But we do need better monitoring and management of the open source code that’s out there. More testing before it goes GA, maybe? Yeah, the Cadillac open source projects might have the money and time to do that, maybe, but take Log4j as a perfect example: It’s two guys maintaining it in their free time and trying to do right by the community. How can they possibly keep up?

    I used to think that the answer was in the repos. My friends at Sonatype had a component firewall that would not let you download an old or vulnerable version of code without warning. That was fine if you were downloading it for the first time. But if it was already embedded—then what? Can we scan for older components in an app? My friends at JFrog can do something similar and more with Xray. And Sonatype’s Nexus platform also allows current and prospective users to scan older apps for vulnerabilities—including scanning for Log4j, absolutely free. But is that enough?

    All of these questions involve open source components. What about third-party SaaS apps? Often, we don’t even know what components or code are in those SaaS apps. Again, it is a leap of faith we are making when using them. 

    These supply chain security issues are a problem that we are just beginning to grasp. And while 2021 has been the year the stuff hit the fan and Log4j may be remembered as the biggest of the first, I, for one, am afraid to think about what else is coming. 

    I do know that the only way to tackle this and any future supply chain issues is if developers, DevSecOps, DevOps, SREs and our security incident response teams all work together. Just like most security issues, supply chain security is all of our responsibility.  I hope Log4j hasn’t bitten you too hard, and that you are on top of your exposure. But let’s start thinking about the next one and how we can better prepare for it.

  • Overcoming Challenges to Automating DevSecOps

    Overcoming Challenges to Automating DevSecOps

    In the last few years, DevSecOps has been widely adopted among organizations looking to get proactive with their security. Traditionally, development teams would continuously implement and deploy new applications into the enterprise and security was an additional bolt-on at the end. 

    However, as the threat posed by cybercrime has increased, organizations have been ‘shifting left’ with their security and weaving it into the beginning of the development life cycle. 

    This offers significantly increased security benefits and means software vulnerabilities can be identified and remediated before they are exposed to the enterprise. The one drawback is around the additional time this takes. As a result, many organizations are also looking to automate their DevSecOps processes. 

    Automation is a critical feature of DevSecOps as it improves efficiency, cuts costs and can streamline operations. However, to get the most out of DevSecOps automation so it delivers real value to the organization, there are several challenges that must be considered first. But what are these challenges and how can they be overcome?

    How to Approach DevOps Security Automation

    When you first move to security automation, you’ll find some security tasks that can easily be automated and others that are harder to fit in. DevSecOps teams will need to use many tools to cover all their bases, but no one wants teams checking lots of different tool outputs. Making it easier to see what’s going on and where risk lies by consolidating security tools and results in a central platform is the way to go. This makes life easier by giving a single pane of glass. Everyone generally agrees that DevOps teams must adopt cybersecurity best practices, but testing takes time. Development teams don’t have time to learn, use and check lots of security tools on top of their existing workload. Automated solutions get to work without any real need for maintenance or management. These solutions trigger tools to work at the right time depending on the outcomes found without any manual input needed. This gives you and your team valuable time back to focus on other important matters.

    Fitting DevSecOps Into Secure CI/CD Pipelines

    The goal of CI/CD pipelines is fast delivery of build and release steps, typically through automation. A key goal of DevSecOps is to alert someone to a new problem as early in that automated process as possible. You need to develop solutions that don’t overwhelm CI/CD pipelines, yet allow flexibility for differing tech stacks, security tools and environments. What you need to ask is, have the latest changes caused any new and important security issues that should be reported before going live?

    Doing DevSecOps Without Constant CI/CD Changes

    Trying to manually run security tools on your entire application source code each day can be time-consuming and hinder your ability to keep up with daily changes. For security checks to keep pace with code delivery in a CI/CD environment, security automation is a must. Security products need to integrate into the development pipeline and enable both the development and security team to work together instead of just throwing things over the fence to each other. 

    Focusing on Issues Raised for Faster Remediation

    All the artificial intelligence and machine learning (AI/ML) in the world isn’t going to remove non-issues better than a manual triage, so here we recommend being resigned to getting stuck. Treat the first run of a set of security tools as a normal security test. Expect hundreds of issues and be prepared to go through them and mark the false positives, duplicates and issues that just aren’t important right now. But do this with the expectation that you’re creating a baseline against which your future pipeline scans can be measured so you’re only alerted to new important issues. By automatically managing and recording the lists of real issues existing in every DevSecOps pipeline run, we can then concentrate on the differences. These new issues need to be flagged and then made easily visible and actionable for all teams.

    Running Multiple Security Tools in CI/CD

    DevSecOps includes setting up many automated security tools to cover the many types of security issues that need to be checked for. Many organizations run multiple security tools. However, as the number of security tools grows and the DevSecOps processes become more complex, many companies then realize the hardest challenge is to make sense of all of the test results within a short period of time. For some companies, this task of bringing together the results from all of these tools is a full-time job, and one ripe for automation. 

    Triaging Security Issues

    False positives and nonsense issues are consistent problems for using security tools. Removing these gives better results and increases the chances that development teams will have a better relationship with DevSecOps processes. Organizations should build a list of issues that don’t need to be actioned and also use automated security tools to flag new issues and differentiate them from the previous backlog of issues. Automating the ‘best fixes’ across all issues also speeds up fix times, while reducing the workload on security teams. 

    Mapping Security to Development Processes

    Dev and DevOps teams are under pressure to get pipelines running smoothly and don’t want security slowing things down or complicating it. By automating security testing in such a way that it can easily handle changes in repos and containers, organizations can overcome this challenge without compromising security. 

    Risk Prioritization

    DevSecOps will increase the number of issues found as well as the speed with which they should be handled. In reality, only a small number of issues will pose a massive risk to the business. Being able to pinpoint the really big issues in real-time helps security know the big risks are being covered and allows developers to focus on the real issues, not lots of small bugs. You can also intelligently determine the changes in risk for quick and automated decisions in the CI/CD pipeline without slowing down the development team.

  • Stacklet Embeds Collaboration in Compliance-as-Code Platform

    Stacklet Embeds Collaboration in Compliance-as-Code Platform

    Stacklet has added collaboration capabilities to its security and compliance platform that automatically groups related notifications, routes them to the right stakeholders and integrates with existing workflows and collaboration tools.

    The Stacklet platform is based on Cloud Custodian, an open source project that provides access to a domain-specific language that enables an IT team to employ YAML files to manage compliance-as-code.

    Stacklet CEO Travis Stanfield said that approach makes it simpler for organizations that have embraced DevSecOps best practices to comply with a wide range of mandates by shifting responsibility for achieving compliance further left toward DevOps teams.

    The latest version of the Stacklet Platform adds communications capabilities that leverage cloud resource configuration and policy metadata to automatically route notifications and escalations. Customizable notification templates add context for teams to collaborate either via email or integrations with Slack, Microsoft Teams and Symphony instant messaging. The Stacklet Platform can also trigger external workflows in tools such as Jira and ServiceNow to track issue resolution and generate reports.

    Ultimately, the goal is to make it simpler for IT teams to prioritize their compliance remediation efforts by keeping everyone involved continuously updated, said Stanfield.

    The number of compliance and security issues that organizations are experiencing in cloud computing environments is especially problematic. Developers often employ infrastructure-as-code (IaC) tools to provision cloud infrastructure. Unfortunately, they typically lack security and compliance expertise, which then results in large numbers of misconfigurations.

    The Stacklet platform is designed to give DevOps teams a way to prevent compliance and security issues using a compliance-as-code framework that spans multiple cloud platforms rather than requiring DevOps teams to employ a different set of compliance tools for each cloud platform on which applications are deployed.

    It’s not clear yet how far left responsibility for compliance will shift. In theory, organizations could avoid penalties if compliance issues are addressed before an application is deployed in a production environment. However, the teams that manage compliance in larger enterprises today are even further removed from DevOps teams than their cybersecurity counterparts. It may be several years before the cultural divide between these teams is bridged. In the meantime, many DevOps teams are taking it upon themselves to manage compliance as code to prevent unexpected issues from arising at the last minute before deployment.

    Overall, the goal is to enable DevOps teams to enforce compliance policies without slowing down the rate at which applications are built and deployed. The hope is that the adoption of compliance-as-code and DevSecOps best practices will address that issue. In fact, as the pace at which applications are deployed continues to accelerate to support digital business transformation initiatives, the number of potential compliance issues steadily increases. More troubling still, the greater the number of compliance issues there are, the more likely it becomes that cybercriminals will find a vulnerability to exploit.