devopsy.kinsta.cloud

Category: Blogs

  • WhiteSource Report Finds NPM Vulnerabilities Fixed Fast

    WhiteSource Report Finds NPM Vulnerabilities Fixed Fast

    WhiteSource today published a report that found most of the vulnerabilities that affect node package managers (NPMs), widely employed to deploy JavaScript applications, are addressed long before they are assigned a Common Vulnerabilities and Exposure (CVE) in the National Vulnerability Database (NVD).

    The report, based on an analysis of the vulnerabilities that WhiteSource tracks in its own vulnerability database and revealed at the AWS re:Invent conference, found more than 90% of vulnerabilities in NPM packages are fixed before the vulnerability is published in the NVD.

    Susan St. Clair, director of product management for WhiteSource, said that fact suggests that, overall, the open source community is staying on top of vulnerabilities as they are disclosed rather than waiting for them to be assigned a CVE number.

    The WhiteSource report comes on the heels of a breach of UA-parser.js, a widely employed NPM based on a JavaScript library that detects browser, engine, OS, CPU and device type/model from User-Agent data. Unknown malicious actors found a way to inject malware into the NPM that modified the library. That vulnerability has since been remediated.

    In fact, the WhiteSource report noted 83% of the CVEs assigned to the most widely used versions of vulnerable NPMs could be remediated before the publication of the CVE.

    In the wake of a series of high-profile breaches of software supply chains, there is a lot more focus on how open source software is actually secured. Commercial software products have a distribution list that they typically notify of a security patch. Open source updates, on the other hand, need to be tracked by developers that use that software. The level of attention individual developers might pay to any number of open source updates can vary widely.

    Many developers are also hesitant to immediately update their applications because they are not sure what impact that additional code may have on their application, said St. Clair. For example, their application may have dependencies that are no longer backward compatible with the updated code, she noted.

    Developers also need to determine the degree to which that vulnerability may affect their applications because they often don’t use every component of an open source library, so the fix being offered may not be required.

    Open source project maintainers often depend on their community to help uncover vulnerabilities and create the fixes as quickly as possible. When a vulnerability is detected in an open source library, the more common practice is to report it to the NVD even after the fix for that issue has been made available.

    In general, St. Clair noted the most important thing for the open source community to remember is to remain proactive when it comes to tracking vulnerabilities and then implementing DevSecOps best practices. The one thing that is certain is that a wide range of malicious actors also are tracking those vulnerabilities closely. It’s not uncommon to see a wave of exploits being created once a vulnerability is disclosed via the NVD.

    The challenge, of course, is that the more open source software that is employed, the more difficult it becomes for individual developers to stay on top of every vulnerability in an application portfolio that only continues to grow. On the plus side, vendors have pledged more than $10 million to help better secure open source software. Hopefully, the return on that investment will become apparent sooner than later.

  • FinConDX: Leveraging DevOps in FinTech

    FinConDX: Leveraging DevOps in FinTech

    The FinTech industry has experienced remarkable growth during the last couple of years as consumers increasingly turn to online shopping and contactless payments. Financial services organizations are adopting DevOps practices to enable faster software releases, improve quality and fulfill customer expectations. On Dec. 9, financial services and tech leaders will come together at FinConDX Virtual Event to discuss the latest and most effective DevOps, DevSecOps and digital transformation practices and tools you need to thrive in the FinTech industry.

    The agenda for FinConDX features a mix of topics, including continuous security and continuous compliance best practices, value stream management, secure development in highly-regulated industries, source code security in financial services, threat detection, incident response, software supply chain attacks, cloud security, emerging technologies and modernization tools.

    FinConDX features a stellar lineup of industry experts, including:

    • Ravi Lachhman, field CTO at Shipa
    • Simon Maple, field CTO at Snyk
    • Dan “Pop” Papandrea, director, Opensource Community and Ecosystem at Sysdig
    • Ravi Yadalam, head of product for cloud optimization at Harness
    • Eric Carter, director of product marketing at Sysdig
    • Maurice Evans, director of product management at BluBracket
    • Jeff Keyes, VP of product marketing and strategy at Plutora
    • Ronen Slavin, CTO and co-founder at Cycode
    • Andrew Fife, VP of marketing at Cycode
    • Edwin Kwan, application and software security, security advisory at Tyro Payments
    • Sanjeev Sharma, SVP, head of Platform Engineering at Trust Bank
    • Colin Wynd, head of Real Time Payments Technology at The Clearing House
    • Mitch Ashley, CTO, principal at Techstrong Group, Techstrong Research

    The virtual environment features five session tracks, live chats, raffles and prices, entertainment and an exhibit hall lined with booths showcasing the latest solutions from leading DevOps companies. In the exhibit hall, attendees will have access to downloadable resources provided by our partner sponsors.

    Registered attendees will also receive a link to access the virtual environment on-demand once the live event is over.

    To see the full agenda and to register, please visit the FinConDX website.

  • A Pro Football Approach to Agile Vs. DevOps

    A Pro Football Approach to Agile Vs. DevOps

    Enterprises and professional football teams have a lot in common. The pressures of being more agile, more resilient while ensuring performance is felt both on and off the field. For CIOs, surviving and thriving requires a serious focus on operations tied to business mandates and growth.

    I am frequently asked questions around infrastructure and operations (I&O) with organizations looking to solve ITOps, DevOps and DevSecOps challenges.

    • How do we shift from legacy ITOps models to transformative digital operations?
    • How do we align process, people and tools for closer integration, consolidation and correlation?
    • How are we structured as teams across functions—ITOps, DevOps, SecOps?
    • How do we transform to analytics-driven operations at speed and efficiency?

    Historically, Agile and DevOps teams face off on the IT playing field as adversaries; both sides want to advance their plans, execute plays and reach the end zone. Disconnects happen when teams are not aligned, goals are not defined and resources are constrained.

    To effectively deliver digital services, teams must focus on the essentials of clear communication, unified culture and measures to ensure resiliency, performance and availability if they are to be effective.

    Traditional I&O teams often operate independently, and knowledge is not shared nor available to cross-functional teams. Attempts to respond and resolve critical incidents can take longer. With DevOps or even DevSecOps, the ideal is for groups to align across functions by breaking down silos of tooling and responsibility.

    In a hybrid ops environment, a lack of integration and coordination between DevOps roles can easily misalign efforts and priorities. Whether it’s hybrid in terms of IT environment or hybrid in support model, there are unique challenges that must be addressed. Not being able to huddle around the same data sets while troubleshooting incidents poses a huge challenge when implementing proper analytics comes into play. Legacy IT operations management (ITOM) tools simply cannot support DevOps’ use of cloud-native services nor support remote IT teams working globally.

    For successful DevOps and traditional Agile teams to coexist and be successful, IT leaders need to change existing mindsets and implement new practices in order to scale.

    • DevOps and ITOps groups have to feel supported by the entire IT organization.

    Often DevOps starts as an experiment with a small group of people working on one application. Everyone in IT, from the leaders to IT workers, needs to understand what the end goals are for integrating new methodologies. Deep realization sets in when DevOps finishes a project and pushes to production which is still supported in traditional ITOps models. It becomes obvious that operations needs to adapt to frequent change.

    • Plan and implement DevOps practices to avoid disruptions.

    With legacy systems still in practice for delivering digital services, DevOps teams cannot disrupt critical day-to-day operations by driving new support models without taking into account migration considerations. Instead, teams will need to work in a hybrid model using application development delivery methods.

    • Collaboration and planning tools are essential.

    The use of collaboration and planning tools alongside automation capabilities is essential in hybrid organizations. Start by automating alerts to generate tickets for the appropriate team(s) to handle versus having Level 1 support process all alerts independently.

    • Build or buy scalable data analysis systems.

    These systems will aggregate and correlate data across infrastructure and digital/cloud-native technologies to deliver contextual insights to problems and provide corrective steps to resolve them.

    Scaling and modernizing operations and creating agile practices is an ongoing process that needs constant planning, testing and adapting. Streamlining DevOps and ITOps processes using automation starts the process as teams begin to work together versus working in traditional silos.

    For winning football franchises, dynasties weren’t built in a single season; the same is true for your digital ops transformation. It takes challenging the status quo—taking a hard look at the deeply embedded ways ITOps has always been done—and reimagining the whole process focused on data-driven digital operations.

  • Funding Rounds

    Funding Rounds

  • WTH? We Wanna WFH | DoD Dual-Sources JWCC | More Nvidia ARM Woes

    WTH? We Wanna WFH | DoD Dual-Sources JWCC | More Nvidia ARM Woes

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Working from home is de rigueur, JEDI redux, and more about Nvidia/Arm.

    Survey Says: Devs Stay Out of Office

    First up this week: GitHub’s annual survey of developers has some fascinating data on the workplace (ahem) “new normal.” The pandemic has radically changed devs’ expectations of where they can do their jobs.

    Analysis: WFH is SOP

    As with other roles, Dev people want to continue to work from home. But does this pose a threat to your DevOps culture? Make sure you can do Ops with equal flexibility.

    Paul Krill: Dev productivity is back to pre-pandemic levels

    GitHub’s “2021 State of the Octoverse” research … sees signs that work rhythm is returning to pre-pandemic levels. GitHub also found that 46 percent of developers who worked co-located with teammates now expect to work fully remotely or in a hybrid environment.
    …
    Based on data culled from more than four million repositories and surveys of more than 12,000 developers … JavaScript and Python remain the top languages, followed by Java and TypeScript. [And] GitHub found that development team performance can increase as much as 87 percent when reusing code and as much as 43 percent when using … CI/CD.


    Zeroing in on the WFH angle, it’s Richard Speed:

    Respondents who were in the office either full or part-time dropped from 42 percent before the pandemic to a mere 10.7 percent now. … That’s a lot of empty desks.
    …
    Fans of the ctrl-c, ctrl-v combo will be delighted to note that GitHub found that when code reuse was made “frictionless” at work, developer’s performance increased by up to 87 percent. … It’s all splendid stuff.


    But beware of statistical fallacies, thinks Henrik Ingo—@h_ingo:

    #WorkFromHome may have been more common than anyone realized. [But] is the population of GitHub users/respondents somehow skewed toward pioneers/early adopters? … This might be a case of correlation is not causation.

    Defense Dept. Orders New Clouds

    The DoD issued its latest cloud contract opportunity notice, with the snappy title of “Joint Warfighting Cloud Capability” (JWCC). Reading between the lines of the notice, it all sounds very DevOps-cum-Agile.

    The Pentagon has backed away from its previous insistence on a single supplier, and now wants at least two cloud providers.

    Analysis: JEDI redux, because AWS wants a do-over

    JWCC is the follow-on from the aborted JEDI procurement. Amazon Web Services was “disappointed” to have been passed over in favor of Microsoft. Now it appears the DoD wants them to share.

    What can you learn from such a multi-vendor DevOps strategy?

    Jordan Novet: Pentagon asks Amazon, Google, Microsoft and Oracle for bids

    Amazon and Microsoft were the finalists for a single Joint Enterprise Defense Infrastructure. … Microsoft won it in 2019, Amazon filed a protest and the Pentagon canceled the contract in July.
    …
    The new effort [is] known as Joint Warfighting Cloud Capability. … JWCC differs from JEDI because it allows the Pentagon to rely on multiple cloud providers.
    …
    The U.S. General Services Administration … said only two U.S. cloud infrastructure providers, Amazon and Microsoft, appear able to comply with all of the Pentagon’s requirements. … The value of the new contracts is not known, but the Defense Department estimates it could run into the multiple billions of dollars.


    In summary, heed this Anonymous Coward:

    Competing suppliers will try harder than one non-competing supplier. The DoD has had a lot of problems with One-Ring-to-rule-them-all systems like the F-35. … JEDI would have been the same.
    …
    Smaller steps with upgrades while testing/using offers better returns. Done right they can get a better deal by making the companies compete against each other in making sure the products are actually useful.


    But look carefully at the DoD contract opportunity, says bustinbrains:

    They’ll all get contracts. Then DoD will simply use the one that they actually like the best. No one can complain because it’s open-ended with no guarantees.

    Arm/Nvidia Deal on FTC’s Radar

    Last week, we talked about Nvidia’s goal to acquire Arm Ltd from SoftBank, and the regulatory challenges it poses—notably in the UK, home of the chip architect. After we went to press, the other shoe dropped.

    Analysis: Just Give Up Already

    As we said last week, Nvidia can’t be allowed to do this. Not only are ARM chips found in 99% of smartphones, but they’re an increasing fixture in the datacenter—especially places that value “performance per Watt.”

    Richard Waters: What’s next for the tumultuous takeover of Arm?

    Nvidia’s acquisition of UK chip design company Arm from SoftBank has provoked serious opposition on both sides of the Atlantic. … The US Federal Trade Commission had its own worries. China, meanwhile, is waiting in the wings.
    …
    The deal has shone a spotlight on the unique position Arm plays in the chip industry. [Nvidia] offered a guarantee that it won’t block any other companies from licensing Arm’s designs. The offer has fallen on deaf ears, with both the EU and UK ruling it inadequate. … Post-Brexit politics have also come into play.
    …
    At what point might Nvidia and SoftBank call it a day? … A stock market listing [is] the most likely alternative, with the UK a favoured venue.


    How’s it viewed from the inside? Here’s a hint from fivemack:

    I was at ARM when Softbank bought them; it came as a surprise to everybody, because we were absolutely confident that we were unbuyable because anyone interested in us would want to use differential licensing as a weapon against their competitors, and we … were confident that regulators wouldn’t accept that.
    …
    Lots of people went home with a year’s pay check in free money. And not a few people … went home with a decade’s pay check in free money.


    More and more DevOps workloads are running on ARM silicon. Here’s Biophoton:

    The low power expertise of ARM has to be an advantage in the data centre (DC) market. … Plus Linux is available on ARM so architectural lock-in of Intel could be eroded in the DC market.

    The Moral of the Story: You are known by the company you keep


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Lukas Mann (via Unsplash)

  • Digital Transformation Driving VSM Adoption

    Digital Transformation Driving VSM Adoption

    The number of organizations using value stream management (VSM) to bridge the divide between DevOps and the rest of the business is expected to dramatically increase over the next two years. The shift is due, in part, to investments in digital business transformation initiatives.

    A survey of more than 500 executives, IT and business leaders conducted by Dimensional Research on behalf of Broadcom, a VSM platform provider, finds the percentage of organizations employing VSM will rise from 42% today to 70% in the next two years. Nearly three-quarters of organizations (72%) that are not currently using VSM plan to do so in the next 18 months or more.

    The survey also finds that 85% of organizations are collecting some subset of value stream metrics today as part of an effort to deliver solutions faster (56%), improve decision making (55%) and increase transparency (54%).

    Laureen Knudsen, chief transformation officer at Broadcom, said that as organizations embrace DevOps best practices more broadly to drive digital business transformation initiatives, they need more visibility into the application development process. VSM surfaces the insights required to not only identify bottlenecks but also track how changes in application delivery timetables will impact key performance indicators (KPIs).

    VSM traces its lineage back to lean manufacturing methods, which called for each step of a manufacturing process to be continuously measured. As software development has evolved from being a craft to an automated process using DevOps best practices—and as organizations realize just how dependent they are on software—an appreciation for the value of monitoring the impact of missed software development deadlines on the business is growing steadily. VSM platforms automatically collect technical metrics from DevOps tools that are then correlated against the KPIs the organization decides to track.

    The Broadcom survey found more than 96% of respondents’ companies are already in the process of adopting digital transformation. Strategic objectives include increasing efficiencies (70%), improving product quality (67%) and improving customer value (63%).

    Most respondents (92%), however, are anticipating product development challenges for 2022. The biggest product development challenge cited is a lack of visibility throughout the product life cycle (45%) followed by inefficient processes (44%), lack of resources (42%) and enterprise-wide lack of understanding of how products deliver customer value (42%). A full 89% of the business leaders surveyed also noted they continue to struggle with organizational silos between strategic development and operations; 93% noted that better alignment between teams would improve their organization’s overall return on investment (ROI) in software development. A total of 95% said they believed VSM would optimize their product life cycle processes.

    Knudsen said that, in time, artificial intelligence (AI) technologies will increasingly be infused into VSM platforms to enable automated feedback loops between the business and DevOps teams. In the meantime, DevOps teams should be prepared to adjust project priorities based on the needs of the business and to endure a higher level of scrutiny as business stakeholders become more involved in setting the software development agenda.

  • Some Change is Bad

    Some Change is Bad

    Some time toward the end of the year, I start considering the volume of change we, as practitioners, have been under for … literally years, even decades. And I start to try and think of ways to keep you all from drowning in change. And then I write something that I hope helps you all focus on good change, and keeps moving your environment forward.

    This year I’m going to say “consider the reasons for the change.” I will attempt to find examples that do not require mentioning a specific product or vendor, so apologies if these read a bit convoluted as I attempt to illustrate this point without castigating.

    There are three basic reasons that a market or marketers want you to consider a change. There are, of course, variants and cases where it is a bit of this reason and a bit of that, but most demands that we change what/how we are doing things can be lumped into “mostly one of these”. They are:

    1. A great idea that is really a game-changer for some subset of the market.
    2. A shifting of who is in control.
    3. Locking customers in or expanding revenue.

    All of these happen at the vendor/project level, and all of them happen at the market level.

    Number one is best illustrated by any of the waves of server virtualization that have come through. From virtual machines to containers, each has solved problems and changed the way we do business. Notably, I’m only talking about server virtualization efforts here. Most of the other virtualization attempts have really not fit the “great idea that is a game-changer” category, though some have limited usefulness. Personally, I think the new version of storage virtualization will be in the game-changer category but am waiting to see it in use.

    Number two is one to watch out for and is most obvious when big-name OSS projects are forked. It is rarely about the technology being implemented (after all, a fork is a copy of the original; at first it is not at all about the technology), but rather about who is calling the shots. There is a bit of this in every public cloud vendor implementing their own version of container management (though this example is more about lock-in, in my opinion).

    Number three is multi-pronged. If your vendor went from selling you hardware or software to selling you a service, they want that recurring revenue and they want you to be linked into that service so deep that you can’t walk away. The much-reviled 1990s mantra of “embrace and extend” is the poster child for this kind of change. There might be a value-add in there somewhere and sometimes the added functionality makes the move a no-brainer. Often, though, the change is simply to the vendor’s advantage, not their customers. As most major Linux distros have shown, open source is not immune to this one, as most of them are making OSS users the de facto beta testers for their paid offerings.

    None of this is to say that the change in front of you is bad—even if the reason it was originally spawned is bad. It is reminding you that, after a constant stream of change, we’re more open to it than maybe we should be and you should remember the only important thing about new changes—do they make our environment more secure, faster, more agile? Everything else is just noise. Yes, all the cool kids are switching to coding in [insert flavor du jour], but does it offer your organization anything new? Yes, it seems everyone is using [insert current CI/CD flavor], but does it address the weaknesses of your current toolchain?

    You all are nailing it. Keep doing it. And don’t be afraid of change, but make sure you know if that new OSS database system is really new, or just a lot of installing the same code under a slightly different name. Because someone didn’t like the new owners.

  • WhiteSource Adds SBOM Tool That Lists Vulnerabilities

    WhiteSource Adds SBOM Tool That Lists Vulnerabilities

    WhiteSource has added a software bill of materials (SBOM) tool to its portfolio that, in addition to capturing the components of an application, also surfaces vulnerabilities that should be addressed.

    Many organizations are becoming more rigorous about making sure SBOM are attached to every software development initiative in the wake of an executive order issued by the Biden administration that requires federal agencies to employ them. While not every organization builds applications for federal agencies, the fact that the federal government is requiring them in the wake of a series of high-profile breaches has led other organizations to review their software supply chain processes.

    An SBOM facilitates those reviews by making available a formal, machine-readable inventory of software components and dependencies that can be used to track their supply chain relationships and identify dependencies and hierarchical relationships.

    Ori Bach, executive vice president for product at WhiteSource, said WhiteSource SBOM takes the concept a step further by also identifying open source libraries. It will also automatically update the SBOM when changes are made to make it easier to identify unintentional or malicious content being installed during application builds.

    Finally, WhiteSource SBOM also provides remediation guidance that ensures updates won’t break the build.

    Bach said it’s already apparent cybercriminals have significantly increased their effort to compromise supply chains as part of an effort to compromise as many downstream applications as possible. In many cases, cybercriminals are now specifically targeting open source projects that create software that tends to be employed across a wide range of use cases.

    An SBOM in and of itself doesn’t make applications more secure. However, it can provide visibility into the components of an application that can then be used to inform DevSecOps best practices, which organizations are implementing more broadly as responsibility for application security continues to shift left toward developers.

    There are two primary specifications that organizations can use to create an SBOM. The Software Package Data Exchange (SPDX) specification is an open standard for communicating SBOM information including components, licenses, copyrights, and security references. Defined by the Linux Foundation, SPDX is now recognized as the ISO/IEC 5962:2021 international standard. The second is the Software Identification (SWID) format that is today widely used by publishers of commercial software.

    Regardless of the format used to construct the SBOM, Bach said there’s an opportunity to more closely align the increased visibility it enables with the application security goals of the organization building the application.

    The primary reason SBOMs are not as widely used as they might be by now has as much to do with inertia as it does with technology. However, as organizations start to expect an SBOM to be made available, it’s clear application developers and builders will need to find a way to automate the SBOM process as applications are continuously updated. The challenge, of course, is that applications are now being updated with greater frequency—thanks mainly to widespread adoption of DevOps best practices—that now need to be integrated with any tool for creating and maintaining an SBOM.

  • ServiceNow Acquires DotWalk to Gain AI for App Testing

    ServiceNow Acquires DotWalk to Gain AI for App Testing

    ServiceNow today revealed it has acquired DotWalk to gain an artificial intelligence (AI)-infused application testing capability that runs natively on its software-as-a-service (SaaS) platform.

    Joe Davis, senior vice president for platform engineering at ServiceNow, said that testing capability will be made available free of charge to all organizations that build custom applications on the company’s platform.

    The number of custom applications being built on the ServiceNow platform has risen sharply since the start of the pandemic. Many organizations have taken advantage of the low-code tools provided by ServiceNow to build a range of applications that use the company’s SaaS applications as a foundation. That approach enables them to more quickly build applications that drive a wide range of digital business transformation initiatives.

    Davis said ServiceNow is committed to enabling organizations to improve the quality of those applications using a DotWalk testing capability that eliminates the need for developers to build and run tests themselves or for organizations to hire dedicated application testing teams.

    ServiceNow

    ServiceNow decided to acquire DotWalk because the company built a testing platform using machine learning algorithms specifically for the ServiceNow platform, noted Davis. The DotWalk tools also analyze logs in the background to identify and validate error sources that can arise during any application upgrade process.

    While there are a large number of professional developers that build applications on the ServiceNow platform, the company is also trying to encourage more so-called citizen developers to build applications. The challenge is many of the end users building applications lack an appreciation for design, security and what it takes to make sure an application scales as more users employ it. Testing tools infused with AI should make it easier for developers, regardless of experience, to build enterprise-class applications.

    Of course, professional developers will also benefit from AI-infused testing tools. Many developers have a tendency to cut short the application development process as they rush to meet application delivery deadlines. The end result, however, is that an application must be updated almost immediately after it has been deployed in a production environment. Testing tools that make use of AI will enable professional developers to more easily run tests they don’t have to build and then maintain, said Davis.

    One way or another, the number of custom applications built on SaaS platforms will increase as organizations digitize more processes. In the last year, more organizations have become adept at extending these platforms using a wide range of low-code tools that automate many aspects of application development. A significant percentage of those organizations are now also adopting DevOps best practices to better manage the pace at which those applications are developed and deployed.

    In the meantime, it’s too early to say precisely how AI will transform application testing. However, one thing that is certain is that even as the number of custom apps grows, applications will be more thoroughly tested before being deployed in a production environment.

  • DevOps Teams Struggling to Keep Secrets

    DevOps Teams Struggling to Keep Secrets

    A growing number of organizations are suffering security incidents related to exposed secrets in DevOps CI/CD pipelines, according to a recent ThycoticCentrify report.

    The study paints a troubling picture: Only 5% of survey respondents said most of their development teams use the same secrets management processes and tools.

    The incidents run the gamut, from secrets published in the clear in public cloud code repositories to insecure third-party code to vulnerabilities in the organization’s own code or configurations.

    The problem is often made worse by security teams not knowing what to protect because development teams have their own cloud infrastructure or the ability to create internet-facing applications directly without security teams’ knowledge.

    “The reason why we see so many security incidents related to exposed secrets in DevOps is that using hardcoded passwords and keys is the easy path to getting things to work,” said Joseph Carson, chief security scientist and Advisory CISO at ThycoticCentrify. “We need to make it just as easy for DevOps to use secrets vaults so even when secrets get compromised they are no longer usable, valid or expired.”

    Carson said moving away from persistent privileges to just-in-time, on-demand privileges will reduce these types of misconfigurations and attacks.

    “Unfortunately, many developers continue to use hardcoded passwords or static keys that, when compromised, allow an attacker to gain control of the application resulting in either data exposure, ability to change the application configuration or move laterally around the network,” he added.

    John Bambenek, principal threat hunter at Netenrich, said the fundamental problem is that when organizations go to agile and DevOps, they remove all friction to moving code quickly—and that almost always includes removing security.

    “That means security is inherently reactive instead of being embedded into engineering, and that’s assuming security even knows what to protect in the first place,” he said. “It’s the natural consequence of removing security from the equation and expecting that it’ll just work on the backend. That approach hasn’t worked for decades; a JIRA board isn’t going to suddenly fix that problem.”

    From Carson’s perspective, secrets management is the ability to move away from hardcoded passwords or static keys to just-in-time privileges or one-time-use passwords so even when comprised they cannot be used.

    “Many privileged access management solutions that protected privileged access for years have extended functionality to developers to help move the value into DevOps so they can manage credentials for applications, databases, CI/CD tools and services without causing friction in the development process,” he said.

    Approaches like privileged access security helps enable API-as-a-service and provides instant availability of secrets, SSH keys, certificates, API keys and tokens.

    Secrets Management

    Bambenek added the problem isn’t choosing a secrets management process or tool, but rather that they aren’t in place at all.

    “Pick something that will keep keys and secrets out of public cloud repositories that developers will use that allows for quick and easy rotation of keys as the need arises,” he said. “The most important thing is that it’s the same tool everywhere that operates in a near-invisible process for developers. Four decades of software vulnerabilities have taught me that if you rely on the developers to do the secure thing, you’ll consistently be disappointed.”

    One Wrong Step

    Davis McCarthy, principal security researcher at Valtix, pointed out developers are not security practitioners by trade.
    “Developers are building new ideas, and no good idea comes without experimentation,” he said. “Experimenting with a new technology stack may lead to misconfigurations that unintentionally expose data.”

    McCarthy said shared SSH keys between users create a larger attack surface, and users will store secrets differently creating multiple attack vectors.

    If one user’s endpoint becomes compromised and their SSH keys are exfiltrated, it’s difficult to know what the first mitigation steps are. Between insider threats and malware, the wrong step could lead to a breach.

    “Whereas a software engineer may hold great knowledge of secure coding methodologies, web application attacks and encryption, they are still focused on executing their craft,” he said. “Organizations that do not build security into the development life cycle create a systemic risk that leads to failures in best practices.”

    McCarthy added organizations can centralize and embed secrets management into the tools already in use in CI/CD pipelines by creating an inventory, auditing it and frequently rotating secrets to sensitive environments.

    Bambenek said the short answer is to make sure the infrastructure of CI/CD pipelines is centrally managed and controlled without the ability of engineering teams to ‘roll their own.’

    “No security control will remain intact if someone can pick their own tools without any real governance,” he said. “CI/CD processes don’t require engineering teams to manage the backend of the deployment pipeline which means someone can ensure those functions are in place to begin with.”