If everyone starts applying cooldowns, won't it postpone the problem? So now there is a considerable amount of users who are affected and someone from the affected group discovers the infection and reports it.
But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
>If everyone starts applying cooldowns, won't it postpone the problem?
There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.
And if every malware developer worth their salt now introduces code to "wait out" that period of time, we're back to square one.
This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.
The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.
Fundamentally, this is a cat-and-mouse game. But I suspect that "time bomb" techniques aren't economically viable for attackers, at least not with current patterns: current attackers demonstrate "smash and grab" tendencies because they know their access is limited anyways. Attempting to wait out a cooldown exposes them to additional detection risk.
Of course, maybe the attacker profile changes over time. But that's the nature of the game.
Most automated analysis isn't dependent on just behavior, but rather suspicious things in the code itself. You have a popular open source package with files that exist on pypi but not github then that's a big flag, or if a similar package suddenly has some base64encoded garbage that runs through an obfuscated exec call. In other words the simple fact that the project has obfuscated code is enough to flag for further attention.
That said if the only issue is time, researchers will just run their automated analysis through machines with dates in the future alongside their normal tests.
Cat and mouse like the sibling says ;) Like if you start changing system time, I'll keep a log of system time to detect any "jumps" and then "behave normally" if I detect this. Of course I'll run the code that does this through "my obfuscator".
The thing with cat and mouse based on time is that this now became a default. I rather liked my odds when malware authors assumed that the defaults were that dependabot updates right away. If the general consensus online seems to be 7 days, then I'll set my dependabot to wait 10 days, so on average I'll catch even things people report over a weekend. Now that the default is a longer time period, I have to change my time period to be even longer, which actually increases my risk in another way: I'll stay vulnerable to _actual_ vulnerabilities vs. supply chain attacks for longer.
Also as an upstream, if your "coworker" releases a strange package without discussing the changes with the broader maintenance group, you might notice after 3-48 hours, but probably not within the hour unless you happened to be online.
> But if everyone will be delaying updates, won't be there less chances to catch it in time?
No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.
> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.
When you're running from a bear on a hiking trip you just have to be faster than your friend. So just set your cooldown slightly longer than everyone else's cooldowns. The cooldown will give security researchers some time to scan the packages so it's still good.
Most of the malicious ones just curl something in a postinstall script, scanners already catch that. The sneaky ones don't look malicious until they run, and three days may not help.
I think the idea is that it gives a bit of time for the companies which run automated scans of new versions to run through and detect any issues with new versions before users install them en-mass.
Watching language package managers reinvent everything distribution package managers have been doing since the 90s has been as fun as watching crypto people reinvent financial regulation.
The publishing topology is pretty fundamentally different: the entire power (and danger) of language package managers is that anybody can publish, not just a privileged few.
(This cuts both ways: I’d say that distribution package managers have learned valuable lessons about what users actually want from language package managers. Learning is a good thing.)
This comparison is tiresome. Distro package managers are curated, language package managers are not. They're serving completely different use cases; the former is the App Store, the latter is the web.
They are literally solving the same problem, it’s just that distro packages operate on a lower level and thus receive more scrutiny. There have been plenty of examples of poisoned Linux packages, both at the source level and at the package level.
This seems to be primarily an issue with a few specific package management solutions that have suffered SCA vulnerabilities recently, not generaly across the board.
I really hate dependabot making generic security people at work so pushy about updates updates updates. They seem to just be dogmatic about whatever dependabot says, forcing churn even when the documented issues are clearly not relevant. I’m not sure how to handle it politically. I’m convinced that updating so much more often is worse, not better.
> I’m convinced that updating so much more often is worse, not better.
The issue of cooldowns aside (which is about delaying updates, not reducing their frequency): you're going to have the same set of problems when you update, whether you do it frequently or infrequently. The difference is that if you update frequently, you'll have a smaller set of updates (so it's easier to debug) and you'll have more opportunity to report issues upstream and fix them in a timely fashion.
It's the same underlying problem as CI and build time. Most people abandoned the concept of projects that take so long to build you can only do testing once a week, because CI that runs on every PR provides a much better experience. This is the same lesson applied to updates.
My logical read of the situation is that I end up making fewer overall changes if I end up upgrading a dependency once, not thrice, to a specific version. And the changes are their own source of risk.
> The reality is that each update is its own potential security issue
Even beyond security issues: each update is a new opportunity for breakage, not only from bugs in the third-party package, but also from unexpected dependencies on the third-party package's behavior.
I don't know if all ecosystems are as bad as node is, but the node ecosystem has terrible issue severity ranking which makes infosec squeamish for no reason.
Every week or so there's a new High+ "vulnerability" that gets published against our dependencies and I have to go look at it to confirm that it's yet another case of "it's possible for someone to give this dev-only tool a bad regex that would cause the test runner to OOM on that branch".
As a sysadmin I'm in the same boat. I've unfortunately never worked with security folks that seemed to have any sysadmin or dev experience. Whether or not this is universal, idk, and I have no idea what they are teaching in these security courses. But I'm tired of security teams telling me "you need to implement these 230 group policies this quarter" or whatever. They don't seem to grasp the externalities of a request like that and how much fucking work it is to vet, test, deploy, monitor, verify, etc. 80% of the time, they don't even know what they do or if it's even impactful for us.
I’ve mainly handled it by pushing my team to be extremely conservative about what dependencies we take, especially if they pull in scads and scads of transitive dependencies.
This elegantly mitigates three problems in one go: update churn, dependency hell, and supply chain attack surface.
It also, frankly, tends to make the code easier to understand. I’m not a huge NIH person but I do have to say that a lot of packages these days tend to encourage ways of doing things that are unnecessarily complex. More than once I’ve replaced a dependency with homegrown code and reduced LOC in the same commit.
But updates to broken packages are still allowed: if a new version is pushed within the three days, it does not reset the cool-down. You just get a pull request to update to a known-bad version instead.
This makes me think whether npm (and other registries) should apply security requirements based on ecosystem impact. Example a package having millions of downloads can have special security measures enforced.
Higher cost (“Mythos” vs static code analysis) vulnerability scanning prior to successful merge to main branch or deployment as an artifact. As risk increases (popular code->greater exposure potential), increase automated, programmatic scrutiny on subject code to lower residual risk.
(application security and vulnerability management is a component of my work in financial services, thoughts and opinions always my own)
What a state of things where we have to fear installing software, and rely on vendors to scan things ahead of time, because our supply chain is such a mess and our tooling is so incapable of (and uninterested in) protecting us.
What would it take to not fear installing software? This isn't a npm problem, its a computing problem in general. Spaces like this are generally pretty against any sort of restrictions or limitations being put on computers under the name of safety (see Manifest v3)
For libraries, I like the Gnu Affero Public License[1]. If you run the library in software with that license, you have to publish all the source of the entire project that incorporates it.
No corporation could tolerate this, though, so the library vendor can negotiate a commercial license of their software for appropriate fees.
That said, corporations are not going to want to negotiate fees with 100's of vendors over constantly fluctuating dependencies in their software.
This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
Similar to Amazon's Dynamo API, whatever the next big language/ecosystem is needs to be designed around _billing_ and automatic license management for # of deployments, seats, call volumes, etc.
> This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
I don't think this idea is going to go anywhere.
If a package is available for free, on convenient licensing terms, developers will use it.
If you make them pay, many developers will prefer to just build it themselves. Coding agents make that easier than ever.
Buying a package involves a lot more paperwork – it needs to go through procurement – and introduces new risks, e.g. what if the vendor increases their prices
There are potential exceptions – software with really advanced algorithms (e.g. solvers for optimisation problems); safety critical software; software needing regulatory certification (e.g. there are some Australian government APIs they won't let you call unless you've hired an auditor to certify the software you are calling them with, and the relevant government agency has approved the auditor's report) – but those exceptions are relatively rare, and the existing solutions are arguably adequate to handle them
I also think it is different for packaged SaaS applications [0] because there the buyer isn't a developer, it is someone non-technical, and "use a coding agent to build it yourself" isn't within their comfort zone or risk appetite (at least, not yet).
[0] conflict of interest disclaimer: work for a SaaS vendor
Manifest v3's actual motive was so shamelessly transparent that most of us just don't allow the "safety" argument for it to really be entertained. I don't have a suspension of disbelief rich enough to pretend I don't know.
> The default applies only to version updates. Security updates still open immediately, so critical fixes are never delayed.
does this require a real vulnerability report, or CVE? if the package is compromised would they just be able to push a false "critical update" that bypasses this wait?
But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.
This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.
The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.
Of course, maybe the attacker profile changes over time. But that's the nature of the game.
That said if the only issue is time, researchers will just run their automated analysis through machines with dates in the future alongside their normal tests.
The thing with cat and mouse based on time is that this now became a default. I rather liked my odds when malware authors assumed that the defaults were that dependabot updates right away. If the general consensus online seems to be 7 days, then I'll set my dependabot to wait 10 days, so on average I'll catch even things people report over a weekend. Now that the default is a longer time period, I have to change my time period to be even longer, which actually increases my risk in another way: I'll stay vulnerable to _actual_ vulnerabilities vs. supply chain attacks for longer.
No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.
> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.
The majority were noticed by maintainers or third party groups noticing things like releases not tied to a source tag, many rapid releases, etc.
Cooldowns won’t stop everything, but it makes a malicious release significantly more likely to be noticed
Build provenance, maintainer alerts on new releases, tying releases to specific git tags, etc all help.
The men see the tiger, one scrambles to run and the other starts putting on their shoes
"Why are you putting on shoes? You'll never outrun the tiger"
"I don't need to, I just need to outrun you"
(This cuts both ways: I’d say that distribution package managers have learned valuable lessons about what users actually want from language package managers. Learning is a good thing.)
The attack vector is generalized.
The issue of cooldowns aside (which is about delaying updates, not reducing their frequency): you're going to have the same set of problems when you update, whether you do it frequently or infrequently. The difference is that if you update frequently, you'll have a smaller set of updates (so it's easier to debug) and you'll have more opportunity to report issues upstream and fix them in a timely fashion.
It's the same underlying problem as CI and build time. Most people abandoned the concept of projects that take so long to build you can only do testing once a week, because CI that runs on every PR provides a much better experience. This is the same lesson applied to updates.
The reality is that each update is its own potential security issue and with supply chain attacks being all too frequent, it's not a panacea.
Even beyond security issues: each update is a new opportunity for breakage, not only from bugs in the third-party package, but also from unexpected dependencies on the third-party package's behavior.
Every week or so there's a new High+ "vulnerability" that gets published against our dependencies and I have to go look at it to confirm that it's yet another case of "it's possible for someone to give this dev-only tool a bad regex that would cause the test runner to OOM on that branch".
This elegantly mitigates three problems in one go: update churn, dependency hell, and supply chain attack surface.
It also, frankly, tends to make the code easier to understand. I’m not a huge NIH person but I do have to say that a lot of packages these days tend to encourage ways of doing things that are unnecessarily complex. More than once I’ve replaced a dependency with homegrown code and reduced LOC in the same commit.
- Restricting packages with similar names as of popular packages restrict expres because express is a popular package.
- Imposing stricter 2FA checks on accounts of authors of these packages.
- Making sure that published packages don't have vulnerabilities and clear npm audit.
- Alerts in case these packages contain a dependency which is new / relatively new.
(application security and vulnerability management is a component of my work in financial services, thoughts and opinions always my own)
[1] https://news.ycombinator.com/item?id=44434355
The grandparent’s point remains the same, the software ecosystem and its supply chain or however you want to call it is a hot mess.
What would it take to not fear installing software? This isn't a npm problem, its a computing problem in general. Spaces like this are generally pretty against any sort of restrictions or limitations being put on computers under the name of safety (see Manifest v3)
No corporation could tolerate this, though, so the library vendor can negotiate a commercial license of their software for appropriate fees.
That said, corporations are not going to want to negotiate fees with 100's of vendors over constantly fluctuating dependencies in their software.
This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
Similar to Amazon's Dynamo API, whatever the next big language/ecosystem is needs to be designed around _billing_ and automatic license management for # of deployments, seats, call volumes, etc.
[1] https://web.archive.org/web/20260712154038/https://www.gnu.o...
I don't think this idea is going to go anywhere.
If a package is available for free, on convenient licensing terms, developers will use it.
If you make them pay, many developers will prefer to just build it themselves. Coding agents make that easier than ever.
Buying a package involves a lot more paperwork – it needs to go through procurement – and introduces new risks, e.g. what if the vendor increases their prices
There are potential exceptions – software with really advanced algorithms (e.g. solvers for optimisation problems); safety critical software; software needing regulatory certification (e.g. there are some Australian government APIs they won't let you call unless you've hired an auditor to certify the software you are calling them with, and the relevant government agency has approved the auditor's report) – but those exceptions are relatively rare, and the existing solutions are arguably adequate to handle them
I also think it is different for packaged SaaS applications [0] because there the buyer isn't a developer, it is someone non-technical, and "use a coding agent to build it yourself" isn't within their comfort zone or risk appetite (at least, not yet).
[0] conflict of interest disclaimer: work for a SaaS vendor
Sandboxing and auditing built into the software from the start. Browser Extensions solved this ages ago.
does this require a real vulnerability report, or CVE? if the package is compromised would they just be able to push a false "critical update" that bypasses this wait?
> Only advisories reviewed by GitHub trigger alerts.
From https://docs.github.com/en/code-security/concepts/supply-cha...