devopsy.kinsta.cloud

Tag: NPM

Why Software Supply Chain Security Is Moving to the Gate

March 2026: malicious versions of axios get published directly to npm. May 2026: attackers forge valid provenance for 42 TanStack packages on npm, with 84 malicious versions shipped before detection. August 2026: a worm uses one maintainer’s stolen credentials to self-propagate through keyv and its dependent packages.

Builds pull in thousands of open source packages from public registries without a second thought, and attackers are getting very good at poisoning these packages at the source. As dependency installation increasingly runs through agents rather than engineers, these supply chain attacks are going to happen faster and more frequently.

The software supply chain security ecosystem seems to have converged on the same fix within the last year: proactive enforcement. npm 12 has disabled dependency install scripts outright by default, closing what GitHub called the single largest code-execution surface in the npm ecosystem.

Repository managers reached the same conclusion. Sonatype’s Repository Firewall can quarantine and auto-release packages ahead of consumption. Cloudsmith runs cooldown policies that hold newly published packages until they have been checked. Socket inspects packages for behavioral indicators of compromise ahead of installation.

Perhaps the most significant architectural adjustment we’ve seen is that JFrog is deprecating the Block Download feature in Xray between April and November 2026. Thousands of organizations worldwide run their software supply chain through JFrog’s platform, so a change to how Xray behaves affects a large share of code actually shipped.

Xray keeps scanning and flagging vulnerabilities, but it loses the power to act on what it finds. Curation, licensed and priced on its own, per seat, evaluates a package’s metadata, including its age and publisher and blocks it at the gate if it fails policy.

Set the commercial questions aside and look at the architecture. A move such as this demonstrates the widely shared conclusion that enforcement belongs at the gate, evaluated before a package downloads, not scanned for after it has already been installed. It is a clear signal about where the whole category is going.

The question it leaves open is more interesting, and it is the one renewal conversations should actually be about.

Everyone Will Enforce at the Gate: What’s Left to Decide?

When gate-based enforcement becomes table stakes, the differentiator narrows to two architectural decisions beneath the enforcement point: who owns the policy decision, and where does it physically run?

These questions matter because the timeline has changed. Maintainer-account takeovers and self-propagating worms already compromise packages faster than catalog updates or ticket queues can move. The next wave of consumers pulling those packages is autonomous agents installing dependencies with no human in the loop, at whatever speed their compute allows. Both questions lead to the same conclusion: enforcement has to live on the wire, evaluated the moment a request is made, and the policy behind it has to be something the customer controls and can change in seconds.

Who owns the security policy? In JFrog Curation, policy lives inside JFrog’s own catalog and UI. Cloudsmith takes a more open approach, letting customers write policy as Rego, genuine policy-as-code, with the code executing inside Cloudsmith’s hosted platform. Either way, the vendor’s platform has to be there for the policy to run: JFrog holds the policy itself, Cloudsmith holds where it executes, but neither lets you take the policy elsewhere.

Where the decision executes is a separate question. Socket, Sonatype’s Firewall Pro, and Veracode’s Package Firewall all answer the question differently: front whatever registry a customer already runs, no platform migration required. But those decisions call out to a SaaS API before a build can proceed. Neutral about which registry you use; but not the same as portable if it depends on the vendor’s cloud.

Portability also breaks for a reason that has nothing to do with outages. Post-merger and post-acquisition, it is common for a large organization to find Artifactory running in one business unit, Azure Artifacts or Nexus in another, and GitHub Packages somewhere else again, not through any deliberate choice. A policy engine welded to one platform enforces consistently on the traffic that platform sees, and leaves a different, weaker posture everywhere else. Universal compatibility, the same policy applied across whatever registries an organization runs, should itself be table stakes, the only way a security policy can mean what it claims to mean.

So we see two trade-offs across the software supply chain security ecosystem. Deep tooling in exchange for platform lock-in. Registry neutrality in exchange for external cloud dependency.

Both trade-offs carry a third cost: developer friction. Developer friction and developer buy-in are two sides of the same coin. JFrog’s own release notes cite CLI timeouts and silent failures as a reason for its own product change. When a vendor admits that a security control developers cannot predict is a cost worth fixing, that’s something to listen to.

Policy written as code and kept in Git is an ownership question, but it is also a friction question. A change a developer can read as a pull request is one they can trust, while a change that arrives as an opaque update is not. Shipping a new policy in report-only mode before it blocks anything is the same instinct in a different form: nothing kills a developer’s willingness to work with a security control faster than a rollout that breaks a build.

JFrog, and others, are moving blocking out to the edge, to the point where a request first meets a registry, before anything downloads. That is exactly where Varnish already sits, and has sat for two decades of caching traffic at the edge of other people’s infrastructure. Varnish Artifact Firewall is the position in this market that keeps both the policy decision and the place it runs with the customer, the direct consequence of taking the two questions above seriously.

Policy is written as Git-native YAML, with a last-known-good fallback if the policy source itself becomes unreachable. It sits in front of the registries a customer already runs rather than replacing them, so adopting it is not a migration, and it works across whichever combination of registries a given organization happens to have accumulated. It is stateless and self-sufficient enough to deploy and scale without professional services, and because it runs in-process, the decision happens before a package downloads, with no round trip to distant clouds.

Quarantine is the mechanism underneath all of this. A newly published version is held back for a set window before anything is allowed to depend on it, hidden rather than deleted, giving time for a problem to surface before a build can reach it. What happens next depends on what the request asks for. An unpinned request for the latest version resolves to the newest version that has cleared quarantine, so a build that never depended on the held-back version keeps working without ever knowing it existed. A pinned version is designed to pass through as a deliberate choice, not silently overridden. A request that fails policy outright, not just time-held, is denied, and returns a clear error and an audit entry rather than a silent failure. Before any of this reaches enforcement, a policy can ship in report-only mode, logging what it would have quarantined or blocked without acting on it, so a team can see the real impact before switching it on. Each of those outcomes, held, resolved, denied, or simply logged, is visible to the developer at the point it happens, which is what developer control looks like in practice

Attacks almost never carry a CVE while they’re still active, the community catches them first, flagging malicious versions on databases like OSV.dev within hours to days of publication, well before any CVE gets assigned. Varnish pulls updated rules from OSV.dev automatically, every hour, so a quarantine window is built on a source that is already catching the important details. Other ruleset sources are being added to widen that coverage further.

What This Means Before the XRay Deprecation

The case that JFrog makes for gate-based enforcement is effective enough and shows that it is no longer a competitive differentiator but a baseline. What is left for a customer to decide is not whether to enforce earlier, but who writes the rule and whose infrastructure it runs on, and the risk driving that decision is only accelerating. A deprecation deadline is forcing that decision for Xray customers specifically, but the underlying choice is the same one facing anyone in this category: keep paying a vendor, per seat, for a decision made on their platform, or own the policy, and the infrastructure it runs on, yourself.

https://www.varnish-software.com/

September 24, 2026
npm v12 Shuts Down a Popular Malware Trick — But the Threat Isn’t Going Away

For years, one of the easiest ways to sneak malware onto a developer’s machine has been to hide in plain sight. Install a package from npm, and any lifecycle script bundled with it runs automatically — no review, no approval, no second look. Attackers have known this for a long time, and they’ve used it well. A poisoned package with a malicious “postinstall” script doesn’t require a developer to do anything other than run npm install.

npm v12 changes that. Lifecycle scripts no longer execute automatically during installation. Developers now have to approve them first. It’s a meaningful shift, and one that security teams have wanted for a while. But according to a new analysis from Checkmarx, closing this door doesn’t close the case. It may just move attackers toward a different target: runtime.

“It’s a bit like locking a door in a glass house: robbers will just start breaking windows,” said Darren Meyer, security research advocate at Checkmarx. “Malware authors often use lifecycle scripts because they’re nearly guaranteed to execute. Infecting the modules in a way that works at runtime requires a little more effort. But with npm’s changes now ‘locking the door,’ the authors will almost certainly find the additional effort worth it.”

That’s the core tension behind npm v12’s new protection. It raises the cost of the easiest attack path, and that’s genuinely useful — attacker cost is one of the few levers defenders actually control. But it doesn’t remove the incentive to compromise packages in the first place. It just pushes attackers to hide malicious behavior inside code that runs later, when a package is imported or used, rather than the moment it’s installed.

There’s also a second problem, and it’s a human one. npm v12 puts developers in the position of approving or denying lifecycle scripts before they run. In theory, that’s a checkpoint. In practice, most lifecycle scripts are harmless, and developers know it. Meyer expects that familiarity to work against the protection over time.

“Developers are under immense pressure to ship quickly,” Meyer said. “Since most package lifecycle scripts are safe, developers will get very used to approving execution of them without doing any kind of inspection first. Even if they do give a cursory inspection of a script before approving it, they aren’t likely to have the time or desire to analyze it well enough to notice cleverly hidden malicious content. This will likely lead developers to approve prompts without reading or figure out how to auto-approve everything they install.”

That’s approval fatigue, and it’s a familiar story in security. Any control that depends on a person clicking “yes” repeatedly, correctly, under deadline pressure, tends to degrade. The fix isn’t asking developers to be more careful. It’s building policy around the assumption that they won’t always be.

Meyer’s advice for security teams is to treat allowlisting as a shared decision between security and engineering, not a blanket restriction handed down from above. “Every team’s risk tolerance will be different,” he said. “But ideally, teams should find ways to work with developers to set appropriate allowScripts policy that results in denying lifecycle scripts that aren’t essential, and approving those that have been assessed to be acceptably safe. Combining this with configuration management that makes ‘strict-allow-scripts’ the default npm behavior can help reduce alert fatigue while still providing a safety net.”

That’s a more practical answer than telling every team to run –ignore-scripts and call it done. Some packages genuinely need their install scripts to function. The goal isn’t zero scripts — it’s fewer unreviewed ones, enforced by default rather than left to individual judgment calls in the middle of a sprint. Meyer’s point about balance matters here too: developers need to move fast and pull in open-source packages without friction on every install, and security teams need some assurance that what’s running hasn’t been rubber-stamped without a look.

None of this makes npm v12 a wasted effort. It raises the bar for anyone trying to compromise a developer’s workstation, a CI pipeline, or a production environment through a malicious package. That’s real progress, and it deserves credit as such.

“I applaud npm taking this ‘lock the door’ step,” Meyer said, “but organizations need to understand that locking this door doesn’t stop malware authors from getting into the house: it just makes them break the window.”

That’s the framing worth carrying into any conversation about software supply chain security this year. Controls like this one are worth implementing, but they’re not a finish line. Attackers adapt to whatever gets harder, and the practical response for most teams is the same one that’s worked for other classes of security controls: pair the technical fix with a policy that assumes people will get tired of clicking “approve” and keep watching for the next window.

August 20, 2026
npm v12 Is Coming in July — Here’s What Developers Need to Do Now

For years, running npm install meant trusting that whatever code got pulled in would behave itself. That trust was often misplaced. Starting in July 2026, npm v12 changes the rules. Install scripts won’t run automatically anymore. Neither will dependencies be pulled from Git repos or remote URLs. All of it becomes opt-in.

This is a direct response to a wave of supply chain attacks that have hammered the JavaScript ecosystem over the past year. In September 2025, attackers hijacked 18 popular npm packages — including debug and chalk — libraries found in virtually every Node.js project. With combined downloads exceeding 2.6 billion per week, it was one of the largest npm attacks in history. In 2025 alone, attackers published nearly 455,000 malicious npm packages. The attacks haven’t slowed down — the March 2026 Axios compromise weaponized one of npm’s most-downloaded packages through credential theft.

The ecosystem needed a structural fix, not just better scanning tools. npm v12 is that fix.

What’s Actually Changing

npm v12 introduces three security-related changes to npm install. First, allowScripts defaults to off — npm install will no longer execute preinstall, install, or postinstall scripts from dependencies unless they are explicitly allowed in your project. This includes native node-gyp builds. Second, –allow-git defaults to none — npm install will no longer resolve Git dependencies, direct or transitive, unless explicitly allowed. Third, –allow-remote defaults to none — npm install will no longer resolve dependencies from remote URLs, such as HTTPS tarballs, unless explicitly allowed.

These aren’t minor configuration tweaks. They close attack vectors that defenders have complained about for years. The Git dependency change is particularly significant. It closes a code-execution path where a Git dependency’s .npmrc could override the Git executable, even when developers thought they were protected by –ignore-scripts. That flag gave many teams a false sense of security.

Why This Matters Beyond the Security Team

Most developers think of supply chain security as someone else’s problem. That’s changing fast. As recently as late May 2026, Microsoft Threat Intelligence uncovered an active supply chain attack involving malicious npm packages registered under organizational scopes that mirrored real internal corporate namespaces. These weren’t obviously suspicious packages. They looked like internal tools.

The average npm project isn’t pulling in a handful of dependencies. The average npm project pulls in 79 transitive dependencies. Any one of those can carry malicious install scripts. Under the current default behavior, those scripts run the moment someone runs npm install. In v12, they won’t — unless a developer explicitly says they should.

Mitch Ashley, VP & Practice Lead, Software Lifecycle Engineering & AI-Native Software Engineering at The Futurum Group, says the change reflects a fundamental rethink of how trust works in the ecosystem. “Package installation is shifting from implicit trust to explicit allowlisting at the ecosystem’s default layer. Disabling automatic script execution, Git resolution, and remote fetches by default concedes that detecting malicious packages after publication cannot keep pace with how fast they appear. Engineering and platform teams now own a dependency allowlist committed to source control, and CI must fail builds on unreviewed install scripts. Verifying what executes at install time becomes a standing engineering obligation rather than a security team afterthought.”

How to Prepare Before July

The good news is that you don’t have to wait for v12 to drop and break your builds. All three changes are available as warnings in npm 11.16.0 or newer, so teams can prepare before the upgrade.

Here’s the recommended workflow:

First, upgrade to npm 11.16.0 or later — this enables advisory mode, where warnings surface but nothing breaks yet. Then run npm approve-scripts –allow-scripts-pending to see which packages have scripts that aren’t yet covered by your policy. Approve the ones you trust, pinned to the version you’ve reviewed, and commit the updated package.json to source control.

Consider running npm approve-scripts –allow-scripts-pending in read-only mode first — it lists every package whose scripts aren’t yet covered without changing anything. That gives you a full picture before you start approving.

One thing to avoid: don’t approve everything with a blanket flag just to make warnings go away. That defeats the purpose. The point is to know which packages are running code on your machine and make a deliberate decision about each one.

For teams relying on internal HTTPS tarballs or monorepos that pin packages to Git branches, the right fix is to migrate to a proper registry — GitHub Packages, Nexus, or Artifactory — rather than adding –allow-remote to every CI script.

Package Maintainers Have Work to Do Too

If you maintain packages that use install scripts, your downstream users won’t get those scripts by default in v12. Two things help: document it by adding a note to your README linking to npm approve-scripts, and reduce the need for scripts by shipping prebuilt binaries using tools like prebuild, prebuildify, or node-pre-gyp, or by moving setup into an explicit command users run after install.

This isn’t just courtesy. If your package silently stops working for users after they upgrade to v12 and they don’t understand why, that’s a support problem and a trust problem.

The Bigger Picture

npm v12 reflects a broader shift happening across the software supply chain. The era of implicit trust — where running npm install was treated like a safe, routine operation — is ending. Explicit allowlists, version pinning, and continuous review are becoming table stakes.

The Shai-Hulud worm, Glassworm, and the string of supply chain attacks throughout 2025 and 2026 have proven that implicit trust in package installation is over. npm v12 doesn’t solve every problem in that space, but it removes the easy path that attackers have relied on for years.

July isn’t far away. The time to run npm approve-scripts –allow-scripts-pending is now.

June 11, 2026
Malicious NPM Package Gets Downloaded 50K Times Before Discovery

A malicious package downloaded approximately 50,000 times from a node package manager (npm) is providing an object lesson for adopting more DevSecOps best practices.

Security researchers from Tenable discovered a “ambar-src” package that was first published Feb. 13 and then updated again before being discovered. It is aimed at developers building JavaScript applications on Windows, Linux and macOS hosts.

More troubling still, in addition to making use of a naming convention that is similar to a legitimate “ember-source” package, the malicious package was also crafted to evade detection. Specifically, the malware abuses npm’s preinstall script hook, a script that executes during the package-installation process. That preinstall script is a file that is part of the npm package itself that also contains code that launches an operating system-specific single line of code to initiate the full infection chain.

The malware that winds up being embedded then uses well-known web services to set up a command-and-control relay through which data is exfiltrated. That technique makes the traffic look legitimate so it is less likely to be blocked by network security platforms.

Ari Eitan, director of cloud research at Tenable, said the one-liner is hex-encoded to make detection and analysis of the malicious code harder. As such, software engineering teams that discover this malicious package in their software supply chain should rebuild their IT environment on the assumption the existing code bases and underlying platforms have been compromised, he added.

Additionally, software engineering teams should also take this opportunity to ensure they have full visibility across their software supply chains to ensure that only trusted packages are being used to build applications, noted Eitan.

In the wake of a series of attacks against software supply chains that, for the most part, have involved some type of malicious npm package, more software engineering teams are hopefully revisiting DevSecOps workflows. A recent Futurum Group survey finds 39% plan to increase spending on software security testing, followed by additional investments in securing application programming interfaces (APIs).

Of course, some software engineering teams may decide that JavaScript packages downloaded from repositories are just too risky. The better part of valor may be to move to another programming model altogether.

Hopefully, there will come a day soon when AI agents are able to identify and remediate vulnerabilities created by AI coding tools, but in the meantime many DevSecOps teams are simply being overwhelmed by the amount of code that needs to be scanned and remediated, especially as AI coding tools download code from repositories without much regard for quality.

Unfortunately, not nearly enough application developers are cognizant of how they might inadvertently add vulnerabilities to a code base. If a vulnerability is discovered just prior, or worse yet after, an application is deployed the cost of resolving those issues is considerably higher. In fact, all the investments made in DevOps tools and platforms to accelerate deployment can easily become all for naught if the code being created only serves to increase the amount of rework that needs to be done.

February 26, 2026
North Korea’s Lazarus Group Targets Developers, Supply Chain

North Korea’s notorious Lazarus Group is using an advanced malicious implant to target cryptocurrency wallets and spreading it via legitimate GitHub profile and possibly through npm packages.

The ongoing campaign, dubbed Operation Marstech Mayhem, is the example of a threat group using open-source code repositories like GitHub, npm, and Python Package Index (PyPI) in software supply chain attacks to spread their malicious code, hoping that developers will inadvertently include it in their software and spread it to downstream users.

It also “exposes a critical evolution in the Lazarus Group’s supply chain attacks, demonstrating not only their commitment to operational stealth but also significant adaptability in implant development,” according to researchers with SecurityScorecard, who have been tracking the campaign over the past few months.

The JavaScript malware includes multiple obfuscation techniques and a novel command-and-control (C2) that operates on unusual ports and is hosting unique Node.js Express backends, which is a break from previous similar Lazarus campaigns like Operation 99 and Phantom Circuit.

An Evolutionary Step for Lazarus

SecurityScorecard researchers last month wrote about Operation 99, a fake recruitment scheme aimed at developers looking for freelance jobs in the crypto and Web3 sector. Operation Marsetch Mayhem is a step beyond, they wrote in a report.

“This divergence in operational methodology not only complicates detection but also suggests that the Lazarus Group is continuously refining their techniques to exploit vulnerabilities in modern software supply chains, including the targeting of cryptocurrency wallets and tampering with browser extension configurations,” they wrote.

The latest version of the implant is Marstech1, which the researchers wrote was embedded in the code of a GitHub repository associated with a profile called “SuccessFriend,” which they believe is Lazarus Group’s profile and is known to create both legitimate and malicious software.

At the same time, they suspect the threat group also is spreading the malware through genuine npm packages that are aimed, again, at project developers in the crypto and Web3 sector.

A New, Ongoing Threat

The Marstech1 implant first emerged in late December 2024 from a C2 server hosted in Stark Industries. It also cropped again last month, with SecurityScorecard calling “limited targeted attacks on the supply; it has not surfaced elsewhere” since the two occurrences. It has claimed almost 240 victims in the United States and elsewhere.

Marstech1 searches Chromium-based browser directories and multiple operating systems to modify browser configuration files, most importantly the MetaMask extension that is popular for crypto wallets. It then looks for Exodus and Atomic Crypto wallets on Linux, macOS, and Windows systems. The implant will steal sensitive data from target directories and crypto.

Such campaigns are common with Lazarus Group and other threats actors backed by the North Korean government, which uses the month to bypass international sanctions and finance its nuclear and other weapons programs. According to blockchain analysis company Chainalysis, North Korea-based threat groups stole $1.34 billion in crypto last year.

Obfuscation and Evasion

To evade detection when embedded into a software project, Marstech1 includes a number of obfuscation techniques, including Base64 string encoding, random variable and function names, control flow flattening and self-invoking functions, and splitting and recombining strings.

To further avoid static and dynamic analysis, Marstech1 uses a two-step process that includes Base85 encoding and XOR decryption.

The researchers also wrote about anti-analysis code that “employs one-time execution wrappers and console hijacking techniques to complicate both static and dynamic analysis. The one-time wrappers allow critical functions to run only once, immediately nulling the callback afterward so that subsequent calls yield no effect. This prevents analysts from repeatedly invoking or modifying key functions during debugging or automated analysis. Additionally, a self-referential check examines the function’s own string representation, a tactic intended to detect tampering or reverse engineering attempts.”

Operation Marstech Mayhem should serve as a “stark reminder that the landscape of cyber threats is rapidly evolving,” they wrote, adding that the integration of the implant in legitimate GitHub repositories and their “subsequent embedding in trusted software packages [pose] a significant risk to both developers and end-users alike.”

Given the advanced anti-bugging defense and self-modifying code used with Marstech1 that looks to thwart real-time analysis, developers and organizations need to become even more vigilant against malicious behaviors and develop security frameworks for supply chain management, the researchers wrote.

February 17, 2025
Checkmarx Report Details Havoc Caused by ‘Everything’ Package on NPM Registry

Checkmarx this week released a report detailing a package distributed via the NPM registry that cripples any machine that is used to download it.

Dubbed the “Everything” package, a user known as PatrickJS, also known as gdi2290, uploaded a package that has more than 3,000 sub-packages. Each sub-package is designed to split dependencies into chunks and to depend on all publicly available NPM registry packages, resulting in transitive dependencies that disrupt pipelines and exhaust storage space.

Following significant effort, The Everything package has since been removed from the registry, but the perpetrators did create a domain where they showcase their handiwork. Whether the Everything package was created as a joke to be played on developers or has some other malicious vandalistic intent is unclear.

Jossef Harush Kadouri, head of software supply chain security for Checkmarx, said these types of incidents are likely to happen from time to time because there is no process for reviewing packages prior to them being loaded into a registry. In fact, a similar incident previously occurred when a “no-one-left-behind” package by Zalastax that was dependent on every publicly available npm package was uploaded in the registry.

It’s only when an issue is identified that the maintainers of the NPM registry will remove a package. Those maintainers simply lack the resources and tools that would be required to scan every package, noted Harush Kadouri.

Heading into 2024, it will be incidents like these, along with vulnerabilities found in NPM packages, that will serve as an impetus for reviewing the workflows that make up an organization’s software supply chain. For well over a decade, developers have been downloading software components from various repositories with little to no regard for vulnerabilities. As more of those vulnerabilities are exploited, organizations that rely on that software are now requiring providers of applications to provide visibility into the provenance of the components used to construct it. The overall goal is to make it simpler to remediate applications, especially when a zero-day vulnerability is discovered.

In the meantime, organizations would be well-advised to curate a repository of software components that have been vetted. Developers can still readily access those components to build applications, but there should be fewer downstream application security issues if the components residing in that registry have been scanned. If developers are allowed to directly download packages and components from a public registry, it’s only a matter of time before an application security issue is going to arise.

Hopefully, as organizations adopt DevSecOps best practices, the number of application security issues encountered will steadily decline. Each organization will need to determine how far left to shift responsibility for application security toward developers, but one way or another, software supply chains need to be better protected.

Of course, the issue will ultimately be forced by regulations that are becoming increasingly more stringent. Once the first fines are levied, it won’t be long before just about every organization that builds and deploys software will quickly move to limit their liability as much as possible. The challenge and the opportunity now is to proactively secure software supply chains today before any remedy that might be imposed on an organization disrupts application development workflows tomorrow.

January 5, 2024
npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: The npm registry suffers spam infestation, and Microsoft makes Google sad. (more…)

March 31, 2023
Speed NPM Releases and Gain Confidence Using an NPM Registry
Building applications in the NPM development life cycle can be very complex. Let’s review the main factors that contribute to this complexity:
- An endless number of variables: Building applications involves a huge number of variables. Not only are there locally installed dependencies in the package.json (which should use absolute versions for each dependency), but there are also environment variables and global system settings.
- Different environments across multiple builds: It can become impossible, especially in larger projects, to maintain the same environment between builds. Even if the same host is used for both builds, local system changes between builds can have a dramatic impact on the published packages. In addition, having a master-slave continuous integration (CI) process further complicates this problem.
- Lack of trust in your binaries: When you pull an internal package in an application, you need to be confident that the binary you are using has been thoroughly tested. Otherwise, it will be the first thing you’ll be blamed for when their project fails.
In this blog post, I will show you the best practices to apply during the npm build process by using an npm registry to ensure that the same binary passes QA and makes it to the production repository.

Streamlining Your NPM QA-to-Release Pipeline

You need to have the following set up prior to creating your streamlined pipeline:
- A repository manager recommended for binary management.
- An automated build server, for example, Jenkins.
- Staging and release environments to be configured as site hosts similar to the production environment, where automated load and quality tests can be performed as per your QA team’s requirements.
- An npm project.
Stage 1: Software development

During the development process, run an automated QA cycle to ensure that the binary qualifies this stage.

Stage 2: Source code deployment

In this stage, complete the function and merge it to your Git branch. This triggers the Jenkins npm install to build the package.

Stage 3: CI server pulls dependencies

The CI server pulls the dependencies when you run ‘npm install’ command.

Note: To simplify the project, use a Binary Repository Manager to store the exact binary dependencies used in the project.

Stage 4: Deploy the Binaries to the target binary repository

Running the ‘npm deploy’ command causes the CI server to deploy the produced binaries to a target binary repository manager repository.

Note that at this stage, the binary is untested and is tagged as a “snapshot” build. As the package is untested, it should be moved to a non-production “snapshot” or “staging” repository where developers can use these packages at their own risk.

Stage 5: Run a second cycle of automated QA on the package

I recommend performing a second round of automated QA in addition to the development stage to ensure that package passes this stage.

If the package passes the above tests, it qualifies to enter the staging environment. The same package is “promoted” from snapshot mode to staging. After the app successfully meets the intended basic functionality, it can be deployed to the staging environment.

Stage 6: Deploy the package to a staging environment

This is typically where the development split occurs happens.

A common bad practice would be to rebuild the application after it passes QA, producing a new binary that differs from the original package.
I recommend using the same .tar.gz when promoting to later build steps.

Perform any QA scripts, which may differ depending on the project. The tests running in the staging environment should closely reflect the real environment. This can also include beta and manual testing.

Proceed to move the pulled binary “Staging” repository.

Stage 7: Deploy the package to a release repository

Once again, it is bad practice to rebuild the same package using the same source code but in a different environment during this phase.

The intended developers, who will eventually be using this package, need to be sure it is the same package that has passed the entire round of testing. Otherwise, you will be the one to blame if their application fails because of using this package.

Hurrah! Your npm registry is up and running.

The npm registry DevOps pipeline is now simplified by using the same binaries during the entire process. To conclude, these are the best practices that are recommended to follow when creating your npm registry:
- Ensure the exact same binary that was built makes it through the entire process.
- Instruct project developers to provide meaningful names to repositories. For example: “Snapshot” = Untested.
- Do not re-upload binaries that have been tested. Testing can modify the underlying binary and have unexpected changes.
Try creating your own npm registry using my customized projects as an example framework:
— Patrick Russell
August 8, 2018
Why DevOps Needs Cloudified NPM

One of the three “ways”—or principles that DevOps guru Gene Kim articulates as defining DevOps—is amplifying feedback loops. One of the key ways that Ops contributes to this amplification is by implementing rich monitoring instrumentation of operational conditions. The move toward more automated configuration management and the rise of cloud-friendly application performance management suites have gone a long way to providing strong feedback loops. However, network performance monitoring (NPM) is sorely in need of cloudification to become relevant to DevOps.

In one of my previous articles on devopsy.kinsta.cloud, I wrote about the impact of DevOps on IT operations management, and noted that infrastructure management tools have largely lagged behind APM in terms of becoming cloudified and API-friendly. NPM is a case in point. Just to be clear on terms, by network performance monitoring, I’m talking about monitoring that collects metrics such as latency, TCP retransmits and out-of-order and fragmented packets. These four metrics can tell a network engineer a lot about whether the network is the root cause or a major contributing cause to application performance issues. Other metrics and data points that are highly relevant to NPM are interface utilization percentage, traffic flow statistics and, for any apps or services that rely on internet communications, BGP paths. These latter metrics and data sets help engineers locate underlying causes in the network, such as congestion on an interface, large spikes in flows, internet path changes or correlation of performance issues with traffic traversing a particular ISP’s network. Sounds great, right?

The problem is, the way these metrics have been collected is all wrong for DevOps practice. For the last two decades, NPM tools have fallen into two main buckets: free, manual tools such as TCPDUMP, MRTG and WireShark; and monolithic, siloed commercial tools. Nobody will ever dispute the value of free tools, but at a certain point, chasing problems around with manual tools just doesn’t scale, so sooner or later most NetOps organizations have to invest in commercial tools.

Commercial NPM tools were primarily architected in a pre-cloud, pre-DevOps era, when data centers all were inside the WAN, applications were centralized and monolithic and all users were somewhere else on the WAN in a campus or branch office. Metrics collection was performed by appliances via connecting to the span interfaces of major routers and switches at choke points—the junctures of LAN and WAN connectivity.

That era is disappearing rapidly. Most new applications are being built on a distributed basis, leveraging hybrid cloud. With distributed application components, there is a much more complex mesh of communications at play, much of which will cross networks and the internet. Further, in most cloud environments, you simply can’t deploy an appliance—not even a virtual one, since in many cases there simply are no span ports.

The problem with traditional NPM goes beyond a lack of fit with modern application architectures. NPM appliances are designed as vertically integrated systems with a GUI as the only real way to get data out of them. As a result, their APIs are usually at best second-class citizens. This is problematic for DevOps because amplifying the feedback loop means being able to serve meaningful real-world data up across different teams.

The cloud and DevOps gap in NPM is pronounced enough that major analyst firms have taken notice. In May of this year, Gartner analyst Sanjit Ganguli published a research note titled, “Network Performance Monitoring Tools Leave Gaps in Cloud Monitoring.” It’s a fairly biting critique of the NPM space that says, essentially, that the vast majority of current NPM approaches were largely built for a pre-cloud era, and are unable to because the new complexities brought by decentralization and full-stack virtualization. As a result, network managers are left in the lurch when trying to adapt to the realities of digital operations.

What is needed to cloudify NPM and make it DevOps-friendly? First of all, with distributed, hybrid cloud applications, there need to be options to embed NPM metric collection agents in cloud server deployments, either directly on app servers or load balancers such as NGINX or HAProxy. With container management and configuration automation tools, deploying lightweight NPM agents to more distributed points is very doable. By collecting NPM metrics on or very close to application servers, you get the most accurate read on “is it the network?” when app performance issues arise.

Another key aspect of cloudifying NPM is where the analytics happens. Appliances have severe compute and storage constraints. In the era of big data and cloud-scale compute economics, relying on appliances for NPM analytics is beyond outmoded. There are a plethora of open-source big data platforms as well as a growing number of emerging commercial SaaS options for NPM analytics. Today, big data network analytics platforms allow engineers to perform deep ad-hoc analyses on billion-row datasets and get answers in seconds.

DevOps’ mandate for amplifying the feedback loop plus the availability of copious cloud compute capacity means that there is no excuse to not aggressively measure everything, including network performance. Networking teams can and must go beyond config automation to change the way and the scale they gather and share metrics to server and app dev teams. Cloud-friendly NPM is a needed step to bring about Net DevOps as a lived reality in IT.

About the Author / Alex Henthorn-Iwane

Alex Henthorn-Iwane is the vice president of Marketing at Kentik. He has more than 20 years of experience bringing new technologies in networking, security and software to global markets. Henthorn-Iwane leads the global marketing strategy for Kentik and helps the company bring its story and solutions to network-dependent organizations around the world. Connect with him on LinkedIn, Twitter or SlideShare.

November 10, 2016
To Be Continuous Podcast: Have We Forgotten How To Code?

Welcome to To Be Continuous, a show about continuous delivery and software development, hosted by Heavybit members Paul Biggar, founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly.

In this episode, Paul and Edith discuss the fallout after the widely covered “left-pad” incident, and while they agree that these sorts of moments move the entire open source community forward, they wonder: Have we forgotten how to code?

[soundcloud url=”https://api.soundcloud.com/tracks/264605904?secret_token=s-Kv9Ep” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

Edith: Hey, Paul. So, somebody unpublished a module called left-pad and the internet basically exploded. And now it’s been a couple weeks since then, so there’s a lot of different topics to cover in terms of what this means for open source, whether it means that people have forgotten how to program.

Paul: So LaunchDarkly. Or, I’m sorry, the left-pad. I’m obviously looking at my own t-shirt.

Edith: You’re looking at your LaunchDarkly t-shirt?

Paul: The left-pad incident was pretty interesting and one of the things that we did when coming up to this was just, like, looking at all the Hacker News discussions and comments on it.

And there’s wildly varying positions on what this means for open source, what it means for whether people can program, what it means for, like, about the Node ecosystem. So to kick this off, why don’t you give us a recap of the entire incident?

Edith: Yeah. So, in short, a programmer had a module called Kik, K-I-K. Which is also [the name of] a very popular messaging app. I mean, even I’ve heard of it and I’m not a teenager.

Paul: The Kik lawyers contacted the dude. Said, “Please take down your module, change it.” And the programmer said, “No. I basically don’t believe in copyright laws.” So then, the Kik people contacted NPM. The NPM people took down the module called Kik. And the guy, in I’d say, kind of a fit of pique, said, “Okay, you took down my one module, I’m gonna take down all my modules.” Basically, I’m taking my marbles and going home.

So, he took down all of his modules. One of which turned out to be an incredibly popular module called left-pad, which did basically what the name implied. It sorted left pads. And without this in NPM, the quote I read was, “Builds all over the world started to fail very quickly.”

Paul: So as I understand it, left-pad was used by another module, which was used by another module, which was used by something core, or like, this incredibly popular module—

Edith: Yeah, it was basically turtles on top of turtles on top of turtles, and at the bottom turtle was this guy who—

Paul: Took his marbles and went home.

Edith: Yeah, or yank the bottom Jenga, basically.

Paul: Gotcha. And so all the builds started to fail. People’s deploy started to fail. And people got pissed.

Edith: People got pissed because they didn’t even realize that they had this dependency. So what happened was NPM basically republished it. They said that, even though you’re the original publisher—

Paul: So there was a lot of outcry on the internet and then they republished it.

Edith: Just basically said, “You don’t have the right to un-publish this.”

Paul: Gotcha. So, um, but they had an Unpublish button, as I understand it. So he did something that was built into NPM to remove a module.

Edith: Yeah, and since then they have tightened their rules a lot. They basically said that this should be something that’s used extremely rarely, if at all, and not something you can just do capriciously.

Paul: So one thing that’s interesting about the NPM ecosystem, and the Node ecosystem in general, is they’ve had a bunch of things like this. Things that are security-related, or that are drama-related.

And people have repeatedly come out and say, “Oh, no, JS is a tar pit, or ghetto,” or whatever it is that people use to describe poor ecosystems, that normal programmers don’t know how to code, and so on, so on.

And what it looks to me like is, you know, each of these incidences brings the Node ecosystem quite a bit forward. To the point that, I mean Node is relatively new, it hasn’t had 15 years before prime time in the way that Ruby or Python did, or 20 years before it got big, and it’s a real, kind of bottom-up thing.

I mean, if someone built it and there were some steroids and then someone else took over, and then NPM took over, and then. It’s this sort of weird language from the ecosystem perspective. It didn’t have the same advantages that, like, Java had, or that C or C++ had, in order to get to maturity. And so each of these incidences propels it closer to maturity.

Edith: Yeah, I compare it somewhat to when electricity first started coming out. And then you were just working through all the issues that could happen. Like, for example, when a power line went down or a transformer blew.

Like, you work through, Okay, what are our points of failure? Let’s not make that a point of failure again.

Paul: Hmm. So, NPM changed their policies on this basis, right?

Edith: Yep, so now you can’t have an individual contributor be able to unpublish, even if they originally published. Now you have to contact them, and it’s much more stringent.

Paul: Gotcha. The response, despite people being … Well, people were annoyed at their response.

Edith: It seems perfectly rational to me.

Paul: Oh, I think it seems perfectly rational. This is the thing about the in trend. There’s always some set of people who are annoyed at everything you do.

Edith: Well you know, the internet is full of billions of people. I mean, there’s gotta be some sort of standard deviation, or otherwise it would be a very boring internet.

Paul: Right, right, and this is why there’s always haters on Hacker News. No matter what you do. If you do it one way, there’ll be one group of haters, if you do it another way, there’ll be a different group of haters.

And so, with left-pad, there were people who were annoyed that NPM retroactively decided that they had ownership over their namespace, which previously, they had sort of gifted to people in the community, the first people to take the namespace.

Edith: You mean the original thing about Kik?

Paul: So there’s that, but there’s also the left-pad. They took away the sort of autonomy of the left-pad offer.

Edith: I read a good article that basically says part of open source is, if it truly is open source, do you maintain the rights to it? Or is it a community project?

Paul: Hmm. The traditional way of deciding that sort of ownership was the fork. That you could go, and you could fork, and when you forked, you weren’t necessarily entitled to the same name, you weren’t entitled to the same trademarks. When Debian forked Firefox, they called it Iceweasel.

When Jenkins was forked, or Hudson was forked, it became Jenkins. GCC got forked. They all got new names. But what was the fork here, it wasn’t even a fork. It was a republish of the same thing, the same software, in a namespace that the original author didn’t control.

So typically you’d own the URL, you know, if you have a software library, you’d own the URL. And they did own, in some sense, the URL within NPM, or the name within NPM’s space, and NPM, the sort of owners of that namespace, overrode it.

Edith: I think they did what was necessary for good order.

Paul: Sure. I completely agree, I think that was a good decision.

Edith: I mean, so to take this to the logical conclusion, what if you had a module named left-pad, and you decided that, like, as a prank you were going rewrite it to be right-pad? Like, sure, that’s … Is that within your rights? Kind of, but it’s a real jerk thing to do that you probably want to prevent.

Paul: This is kind of one of the dangers of the whole ecosystem around centralized package managers, of sorta trusted package managers, where there’s no actual trust involved. So, what I mean by that is what is published does not have any gatekeepers.

Edith: Yeah, exactly.

Paul: I mean, it’s great, it allows an ecosystem to grow much faster, but it has safety concerns because people generally don’t pin their packages to an exact version.

Edith: You’re basically pinning it to a floating value.

Paul: You’re pinning it to a floating value. And that floating value could change. There’s no guarantee of what a minor number means or what a major number means or what a patch number means or what a patch even means. There’s some sort of, like, general community guidelines in semantic versioning, but there’s no enforcement of that.

Edith: Yeah, and I mean this basically goes back to the whole Cathedral versus Bazaar thing.

Paul: Right, right. It’s very Bazaar. And intentionally so. It’s the thing that allows a community to grow, and is there a point at which open source communities need to, like, batten down the hatches to prevent that sort of behavior? Or prevent that sort of risk?

Edith: Yeah, and it’s really interesting. I think this goes back to something we’ve talked about with Nadia Eghbal and Sean Byrnes a couple episodes ago about when you buy from a corporation, you have some sort of guarantee of standards that they publish somewhere. You know, this is our guarantees versus with open source, you don’t normally have those same sort of guarantees.

Paul: Even when you do, it tends to be—

Edith: There’s no enforcement mechanism.

Paul: Right. So there’s no enforcement mechanism in the Bazaar.

Edith: Well, there’s no, um … Like, so with a company, if a company has said, “Okay, we’re gonna release twice a year and it’s gonna have this level of quality,” and they break that, you can walk with your feet or your money. If an open source project stops or does something that you don’t like, there’s very little enforcement end.

Paul: Theoretically, you could walk. You know, you could fork the thing, but when you end up in an ecosystem that has massive network effects, and in the way that NPM does, you can’t walk with it. Because the pends is all transitive. The left-pad was, you know, a turtle way at the bottom of the stack.

Edith: It was almost like a turtle hatchling.

Paul: So, to fork left-pad, you would have to fork the entire NPM ecosystem.

Edith: Yeah, and it’s not even. The fork is not the issue, though. The issue was that people were depending on it being in a certain place in a certain way. It wasn’t that it wasn’t forked, it was just like this entire structure at the bottom was gone.

Paul: I guess what I’m saying is that if you get pissed off with a certain set of developers or a certain set of practices within the NPM community, because they’re so interlinked, you don’t have the traditional open source option of forking.

Edith: Absolutely. Absolutely.

Paul: You could rename it. You could pick a different library, but because of the way—I’m not sure if they still do this, ’cause I read something about it, but in NPM, it used to be that the version of the software that your library took was the version that library got. Do you know what I mean by that, okay. So let’s say there’s some software, let’s call it left-pad.

Edith: Let’s call it right-pad.

Paul: Right-pad. So, right-pad version one is being used by library A. And right-pad version two is being used by library B. In the Java ecosystem, and many, many other ecosystems, only one version of right-pad would be used.

And I believe in NPM, both versions of right-pad would be used. So library A will use version one, and library B will use version two. Now that’s definitely the way that it used to be, I’m not sure if they’ve actually changed that.

Edith: Yeah, I don’t know, but that sounds very messy, very quickly.

Paul: So both options are terrible. And there isn’t a right way to do it.

So, really, what it depends on. So, the NPM way of doing it, every library get its own version of the dependencies, is fantastic. Because it lets that library do what it actually intended to do under the covers.

If, however, you start having an object that gets sent around, or that two modules are, let’s say it receives a, I think, an HTTP packet or something like that would be a good example. Let’s say you take an HTTP packet from one module and just hand it to another module, and they have different understandings of the shape of that object, or what functions you can run on that object, then that’s messy.

But, on the other hand, if you override the version of right-pad that some other library’s using, you end up with all sorts of problems. This is a major problem in the, sort of the Maven ecosystem. And at CircleCI, we used Clojure, and Clojure is built on top of the Maven ecosystem, and we would have bugs where the library that we used.

One library that we used had a dependency, and that was in the package file for another thing. And so, that’s the version of the packets that we used. So we would have to pick. And as a result of that, we have to pick the library that works. And, often, there isn’t a library that works for both, or there isn’t a, sort of subdependency that worked for both. So those are close to forked no matter which way you do it.

Edith: Yeah, I mean, we ran into the same issues with our STKs. I mean, ’cause we have dependencies on other libraries that sometimes conflict.

Paul: Right, disaster.

Edith: On the other hand, I mean, the bigger point, I think, was. So there was an interesting backlash, I think, that everybody kinda projected what they wanted on the NPM thing.

People said, Oh, people don’t know how to program anymore. You know, why do you even use what they called left-pad? You know, why don’t you just grind your own wheat and make your own bread from scratch and eat it at lunch also?

Paul: I have no sympathy at all for this view of the world.

Edith: Uh, that you should make everything from scratch? Or—

Paul: That, “Oh, we’ve forgotten how to program because we need left-pad.”

Edith: Yeah. The fact that we have modules available that make it easier and easier and easier to program is a good thing.

Paul: I think it’s kinda different to that. I think it’s more of a curmudgeon view of the world. It’s a get-off-my-lawn—

Edith: Wait, that you’re the curmudgeon?

Paul: No, the people who complain, “Have we forgotten to program anymore?” I think it’s a no true Scotsman kind of argument. Real programmers know how to write their own left-pad, real programmers write in statically typed languages, real programmers write in C, real programmers, and I’m going back in time here, real programmers write in Assembly, real programmers hand-code their. Tweak the needles on the.

Edith: They do their own garbage collection.

Paul: Oh, and it’s garbage collection, what is this? Garbage collection is for amateur coders, right? I mean, you could just as easily have written, have we forgotten how to manually allocate memory?

Edith: Or, you might as well say, like, nobody knows how to make their own transistors anymore. I mean, like, the entire computing revolution has been that you no longer have to assemble a computer by hand.

Paul: Right. And each layer of the ecosystem allows a vastly larger number of people to take part in it. So I think the thing that we’re seeing now. This has been happening for five or 10 years, but what’s getting really, really big now is people who only know how to front-type program.

People who only know Javascript and HTML/CSS, and they go to boot camps and they come out, they maybe have some, like, vague real understanding.

And they call themselves coders, and we, the real programmers, who know how to write a quick sort, and who know how to write left-pad, we don’t welcome them to our community. And frankly, I mean, I have felt this in the past, I have since changed my mind, but it’s a very, very common perspective.

This podcast is brought to you by Heavybit, a program that helps startups turn products loved by developers into successful businesses. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.

June 20, 2016