npm is unsafe* 

my life of developer is unsafe

My favourite case of loading js from other websites is still barclays UK loading a js file from an older waybackmachine copy of their own website.

15:30 I think you kinda misunderstood, what many people are worried about. It's not, that known good packages (like react) get compromised, but that you don't really know what many dependencies do. Maybe that small utility package used to add some spaces to the left of your string also has code to send every keystroke to some server and grab the passwords of your user. Nobody is looking at the code of the hundreds of packages many projects include.

The secret: It's unsafe all the way down.

It is not about NPM as such, it is about having too many unreliable dependencies. In C/C++ dependency management used to be hard so historically people tried hard to keep their dependencies few and small.

Just the fact that npm audit has to exist proves you wrong. We cannot keep track of the thousands of left-pad, is-number and similar packages, so we have to build a workaround for it. "Back in the day" you had a database wrapper like pdo, and maybe some Apache modules like Headers, Session etc. and that was it. Easy to keep track of and less points of failure. I think this is an issue with js in general, because so little needed functionality is built in and the trivial code is usually very flawed because of some edgecase. In other modern languages like C# you usually only need 2-10 nuget packages for a large project and even then most are provided by Microsoft. That’s my issue with JavaScript and its ecosystem.

It feels unsafe when I go npm install on a project where my colleagues and I are working on and it installs a multitude of dependencies because we don't just use plain react and react-dom. We use other packages which do have a bonkers dependency graph. That then installs 1500 packages with multiple vulnerabilities which you can't fix because they are multiple dependency levels deep and those packages that depend on the vulnerable packages sometimes depend on specific versions so you can't just go all willy nilly and update all the vulnerable packages because you might break your entire appllication. That is something that feels unsafe

I'm surprised that no one has mentioned that his dependency versions are not "locked", you know that the ^ means that you can automatically download ANY new minor or patch version... so in this example, you can download react 18.4, 18.3.2, 18.5...etc if you actually want to lock the version, you have to remove the ^ its all in the documentation on semver in practice, its also better to set up your own private npm which proxies the public one, solutions like nexus can do this automatically for you, so you can host your packages and still get all the packages from the same URL

Slight disagreement on Theo's take on dev dependencies not being as much of a risk (as regular dependencies). Having to hijack the build step vs being conveniently bundled is a trivial difference for a malicious author.

Your screen at 3:56 shows the thing that really stresses me out about npm; everything you download throws up twelve thousand warnings about how there's something wrong with it. And I'm a noob, what do I know, it usually works anyway and tutorials all act like that's normal, so I learned to just ignore it, but it feels like learning to ignore an overreactive smoke alarm.

I don't think you even understood their worries here. You still remember the xz backdoor, right? I've seen projects relying on random hobbiests' repo. Not that those hobbiests are doing a bad job. The problem is that you have no idea what level those devs' securtiy awareness are at. Offensive operators will focus on the weakest link, and it's a disater when even experienced devs have no idea where the weakest links are at.

It doesn't matter if the npm server itself is "safe." What matters is the number of maintainers of those packages you depend on. At ANY moment, ANY number of them can go full evil mode and start pushing malware into the package they control.

The bundler doesn't distinguish between dependencies and devDependencies. Using devDependencies does not exclude those deps from being shipped to the user. The bundler ships everything to the user that is referenced from your source files. Rather, devDependencies controls what is being removed on "npm prune --production" from your node_modules directory. That is important for node apps without bundling since the require function in commonjs will load js files from your node_modules folder. But if you bundle your apps, you might as well just have everything be dev dependencies, since your dependencies' code will be *bundled* into your build artifact anyway.

Honestly the biggest risk with npm is the near heart attack when you have a typo in a package you npm install and it downloads a package with 19 total downloads

don't try checking the dependencies of react-native on npmgraph 💀

Literally no one thinks npm can somehow pwn your website at runtime, the point is that you're downloading and executing dozens of megabytes of arbitrary code on your machine. You're not safe just because there's no active NPM scandal, literally open any year old project and NPM will automatically warn you about a dozen new security vulnerabilities that were discovered, it's safe to assume there are dozens more that have not been reported yet and are actively being exploited by malicious actors. And why assume devdeps are safe? If a package like eslint got compromised it could trivially inject malicious code into your bundle. Just because it's supposed to only run on your machine doesn't mean it can't still access files in your dist folder.

"Getting always the same thing" is not about security, it's integrity. Unsafe package can pass integrity check and still to be unsafe.

You missed the point by a mile here theo

Dependencies are unsafe. If you care about security you pick a stack without a deep dependency hierarchy.

It’s not only about what’s unsafe for your users. You are basically opening up your file system to unknown entities, so it’s unsafe for you. Whenever you run a script locally, there’s so many scripts that you don’t know about that now have access to your files.

Perspective of a new dev who who just did a basic training course and a few personal projects, so pretty much a baby with not much knowledge yet. The concept of using cdn has something that have felt crazy to me from the first time i heard about it and i never want to use any forein cdn in any of my projects. I do consider package managers due to lock versions which will prevent automatic updating to newer versions with possibly security issue and that is certainly a major step in securing the depencies overall. But the constant fear lurking in the back of my head is that this just protect from the introductions of future security issues, in all cases we are still adding dependencies in an effort to not waste time reinventing fire for every use case. That means we add a lot of foreign code and among that sea of dependencies there may be some code that has been cleverly hidden for malicious use and not found yet. It gives the feeling that a decent chunk of the ecosystem is built on trust, and trust is a very expensive commodity in our era.

Lately all of Theo's takes have been: "actually it's not that bad". About things that are pretty bad. Here, he's not recognizing the fact that the average npm-based application has around 700 dependencies. And nobody who writes an application like that is even skimming through them to have an idea of what they do and why your application needs them. Much less an in-depth code review to check that it's not bundling some keylogger into your application. He's also not recognizing the fact that dev-dependencies have a say on what gets bundled in the final artifact, so they are as much of an attack vector as regular dependncies. At the very least some solid advice would be to avoid taking additional dependencies when possible, unless (1) you don't really care about security (which is fair in some cases) or (2) you're willing to go through the dependecies that you're taking on, at least enough to reasonably assume that they're safe.

Few people know about "npm ci"... That's what this misunderstanding boils down to. Everyone is using "npm install", but "npm install" ignores the lock file, while "npm ci" honors it.

One thing that wasn't quite clear in the video, the 'Initial npm install' the latest version that resolves within the semver range specified in package.json gets installed, i.e. if you have "react" : "^18.2.0" in package.json it would actually install 18.3.1 (Latest version within that semver range as of this date), so the package-lock.json would have 18.3.1 listed, even though package.json uses the semver range "^18.2.0" specified. Subsequent calls to npm install would read the exact version 18.3.1 from the package-lock.json file.

The python ecosystem is Dependency hell. Matplotlib or Scipy have so many dependencies it's insane. If you look at all the ml stuff that is built on top of this stiff it gets even more crazy.

Hate to be harsh about this one, but maybe someone could give Theo a give lesson on how npm actually works before his next video. Cause this video includes alot of misinformation, and does not at all address the valid concerns regarding npm. - npm install - ignores you package-lock file. So yeah, you would get the newest version within your semver range in you ran the command. Hence why we got npm ci. - devDependencies could absolutely be included in your bundle. As far as I know the bundlers just include everything you reference in your code. So unless your team are very strict with devDeps vs deps when installing packages you might very well have included something just listed in devDeps. - no mention of "postinstall" scripts, which is basically a remote code execution whenever you install a package.

The package lock is nice but when you upgrade you can still have this dependency tree that is overloaded and devs just aren't going to review. This happens in most packaging systems of course but maybe has a bigger risk exposure in JS.

Excellent lesson, really well done. Its easy to never learn what package-lock does unless you've experienced problems with not having it or updating it by mistake. I remember pre-lockfile era zipping node_modules and having that be the only way to get the right packages and build. Otherwise one or more semver compatible updates would break something. Nightmare.

Everyone thinks that hacking something is very difficult and technically challenging but in reality all you have to do is offer a free cdns for javascript and even developers will fall for it....

The issue is that even if don't add these files directly to our website, but the bundler still use that code from hundreds of unknown packages. Also even if we are using well reputed package, it might have some dependencies that have some vulnerability or it was never updated to cope up with latest security standards, so it is indeed unsafe.

lock file and hash does not guarantee security; at most, what it guarantees is "reproducible builds" and integrity. You can have a lock file and even a Merkle tree of all the hashes of all packages in a project, and your project can still contain malicious packages.

the problem is not that you host directly from npm... I dont know a single person that got that confused, why did you spend so much time on it? The problem is that even a simple project can easily have hundreds of dependencies and there is no way you can verify that the source code doesnt contain malware or a backdoor, all it takes is one developer screwing up or getting screwed and push a malicious package that gets included by other projects and remains unnoticed, because seriously, a lot of code is unchecked on npm, and you end up with a malicious app either for your users or compromising your own machine. all it takes is a node js file downloading and executing a binary on your system... Dont pretend this is not an issue, there is a very good reason deno the successor to node went with a very strict permission model, where you have to opt in to things like file system access, etc.

i love you theo, thanks for teaching really og stuff. keep doing this type of content

There are some really good points in this video around how to improve your security posture by pinning npm packages. I do think however that since we are frequently upgrading packages and, in many cases, automatically using tools such as renovatebot/renovate that it isn't much better than including a CDN link in your html. Yes, there is more of a chance to that a hacked package gets identified but as we have already seen, it is not always the case, particularly where the maintainer(s) are they ones who introduced the malicious code. There are also malicious packages in npm so we should be cautious about which packages we are using and the reputation of it's maintainer(s).

Go has a decent standard library which is often big enough for basic tasks and services.

I don’t think anyone thinks NPM is downloading assets in realtime. The problem is that you’re often blindly downloading possibly malicious code during the development process, and you have no idea if that’s contaminated your published site, because you didn’t audit the code.

There is a risk but that's for any package management solution. A project or library could be compromised without people knowing - e.g the recently caught SSH compromise thay was luckily caught by an MS dev. This was some malicious code in xz utils - it was almost purely a fluke thay it was discovered so quick. By relying on any third-party library you're creating a potential supply-chain risk, period. NPM would be a risk based on the sheer volume of third-party dependencies. Just because a dependency could have a limited window of compromise, it doesn't make it any less likely or risky you'll end up distributing it. It is a big issue but it's not unique to NPM. It's not even unique to package Managers.

Dependence is unsafe

theo with the low level learning thumbnail style got me a bit confused at first lol honestly a huge fan of this style its so clean imo

Web development is looking more and more like deploying a native application including shared libraries.

Rare hard disagree with the interpretation of the problem. The problem is not the big react package, but the smaller packages that get added over time, with tons dependencies of dependencies. And most projects I see have too many of these small helpers with a ton of baggage. This problem exists in Linux and other software too. And the awareness there increases too. The lock file helps keeping the software running without breaking changes due to small package changes. But nearly no one checks for security after an update to the lock file.

thats why i only use vanilla javascript and css, just rawdogging ajax

doesn't caret automatically use the latest version with the same first number? like if someone just pushed something bad as react 18.3.2 only people with exact version specified will be safe btw I totally thought it's written like carrot lol

Beautiful!!! I'm so grateful for this video for reasons I can't go into. 😅 Thank you for posting. 🙂

The solution for the company i work for is often that we load a version of stuff like polyfill and put them in our own cdn. We have some sites where we was forced to use plain html, css and JavaScript. So this was our workaround to avoid external cdns

I used to think everyone clowns on Theo for trolling purposes, until today 💀

About dev dependencies at 11:30, as far as I understand "typescript" is there to translate your .ts code to shippable JavaScript, so if it get's hijacked it basically can append to your code whatever it wants. May be there are other similar options, so I wouldn't be so sure if it's a good point to care less about devDependencies security. Otherwise I would agree, if you need to include some third party code, package management is the best option, no doubt.

Yes batteries are not included for example you can seed your random generator (i wanted to shuffle a list randomly in a deterministic way for a given user to pagination to work). So I had to get a random generator from npm. But also on the other hand there are packeges like is odd and is even for those I'll happily wear dependency shaming hat.

Something I like about Deno is that the standard library is very comprehensive -- it has a built in test runner, linter, formatter, collections package (ie: lodash), etc. The amount of tooling you need to reach for outside of the Deno ecosystem is much smaller, and the reduced surface area means less opportunities for supply-chain attacks.

The ^ caret before the package version number in both your package and package-lock files meant "Compatible with version", meaning that npm install will update to the latest compatible version (using semver). Does that not apply to the lock-file? As I understand, the only way to properly pin all of your your dependencies to a version would be to remove every single one of those carets.

Great video. Only one thing, it's not because a package is in devDependecies that it won't be in the final bundle. It won't be in the final bundle because it's not imported and used in the frontend code (tree checking by bundlers).

I'm a little confused but think the versioning info Theo mentioned might not be 100% correct. The version "^18.1.2" will use newer version provided the '18.' doesn't change. If one really wants and lock into 18.1.2 the version should be "18.1.2" putting the caret '^' will lock to 18.x.x. And to bind the range one could use ">=18.1.2 < 18.3.9" so 18.3.1 or lower are fine... Sorry if i'm being a bit pedantic i'm thinking if some hacker really wanted to be evil, they could just as easily put 18.1.99 and slip in a rouge version. I'm more a tester at heart and not a developer. I'm only looking at if i'm a hacker how could i slip in a bad version... I might be conservative and assert on the production build build the versions are locked down to one exact/specific version... i.e. I tested and certified the app with these exact version, anything different... It is a completely different product...

I think what people dislike about npm is that prepare scripts can, by default run automatically on installation, so you can't open up your node_modules and "review the code on your machine" because the malware would be a supply-chain attack and it would be located in the prepare script which gets run when the module is installed (examples of this in legitimate cases like how lerna initializes, like how git commit hooks work, etc), so they could cover their tracks theoretically before you knew what hit you. People who use arch linux run the same risk installing any package, it's all abt risk tolerance really when it comes to supply-chain attacks. would you rather things be on the bleeding edge or delayed by a year or two so that things can be safe and approved... (npm equivalent would be like a corporate npm mirror that scans for malware)

thumbnail looks like low level learning

To be even more sure of a package, NPMs new trusted build feature allows packages to be distributed with a build log of how we got to the package you’re installing & other build specific info :)

Honestly I'm surprised that this video didn't mention manifest confusion - since that's one of the big risks of NPM, as far as I understand. Darcy Clarke has a great blog post about it. (and his new company is a new registry)

One key point in here is pinning to versions vs latest. After the solar winds supply chain issue people were STILL downloading the malware’s version AFTER the brouhaha

My experience with node is when I tried a git visualization utility and node downloaded ~4000 dependencies and tried to build them all and broke at dependency 2736 and I decided npm is an infuriating mess and stayed as far as I could from the whole joke of an ecosystem. When people complain of dependency bloat, these are the experiences they're drawing from. Not consummate developers using package locks and appropriately vendoring their dependencies and all kinds of other voluntary best practices that half the community ignores. A lot of times, the requirements of a project compels you to work with a dependency that doesn't have its dependency sprawl nearly as well-checked as react, and just like with reading EULAs, nobody has time to do the due diligence to do a depth-first tree traversal of the dependency graph of everything to ensure they're including is trustworthy. I know that left_pad is ancient history by this point, but it doesn't seem like the fundamental dependency recklessness that precipitated that disaster has been corrected.

People are using tools now knowing anything about them. The biggest security issue is not NPM, but those people.

Package locks don't fix anything, big dependency trees are just inherently unsafe. There could be a piece of malware hiding in a small dependency many layers up the chain which persists across many versions of the package, pinning a single version does not fix this problem. What needs changing is the JS ecosystem's mindset on modular dependencies. Big monolithic dependencies of trusted code is always going to be safer option.

I usually always vet my packages before I install them, which even then is rarely. We typically have such specific functionality that it makes more sense to just implement it yourself rather than relying on packages.

I'm a SecOps engineer, the main concern I have with npm (as someone who uses it for dev work on an in-house security automation app) is more about the dependencies themselves. I try to avoid using dependencies when I can avoid it, even if it means extra work. I also try to only use modules with frequent updates and weekly downloads in the millions. Still, if you're using modules like this, you should always be using a SAST tool like Snyk.

F*king GOOD CONTENT!!! Your awesome at explaining!

Yes you provide the code from your server but how do you guarantee that deps from npm you've provided are safe? What are you on about?

Because you're using the caret (`^`) in your `package.json`, there's still a risk that an exploit could be introduced through minor or patch updates, even with a lock file in place.

thank you for explaining that package json stuff, was useful for me

I was on vacation for 3 weeks so now i'm catching up 3 weeks of Theo videos.

I think people are just talking about not knowing what their dependencies do, period. I have to admit, it makes me uncomfortable too. But the alternative is that all of us need to reinvent the wheel all the time, I guess. So my baseline anxiety just exists a notch above since I started working with dependencies, and that's how it is.

thank you for making this video.

The bigger point in security issue of npm is the "XZ-effect".... look at how tricky and complicated malicious actors hide stuff around and now look at npm.... do you really have any real ability to check your dependencies? I prefer minimal amount of very clean dependencies and no "package management". It really goes the wrong direction...

Having experience with apt package manager I kinda felt relieved when I first opened a package-lock file, there's no reason at all for that tweet

I just download jQuery to the websites root, and yes I still use jQuery, and an outdated version of PHP for my personal site, there is no private user/personal data accessible on it though.

You misunderstood this. With npm, you install a package it grep a lot of packages that grep packages, and you end up with 100 or more, no one read all of the code them, if 1 of them is bad we are done, it is so easy to get bad stuff in, just make left pad like code, and wait for a lot lazy dev to download and you upload a update and boom

Elm packages are probably the safest you can get. Since all functions must be pure, there are no hidden side-effects at all. If a package wants to (for example) do a http request, the developer will surely notice it, since it’s going to be in the function return type and you must explicitly pass it back to the elm runtime to execute it. Pure languages for the win. The Roc language also have a really interesting approach for this problem, but it’s kinda too much to explain in a RU-vid comment ;)

I never heard of anyone being "scared" to host their own JS. The reason folks embedded external JS or did the whole CDN thing was performance. Your website would load just a bit faster if you're user already had the common jQuery, for example, in their cache.

Hope the bundler itself won't get compromised...then it might starts pushing all the dependencies into bundle 😂

10:40 I think lots of people think react has tons of dependencies due to their first experience with React being create-react-app, which visibly installs a ton of dependencies.

"jQuery CND is a thing of legends" he says, as I import jQuery to my flask app because my work blocks use of node and any modern JS frameworks

this moment section appears scared of things they don't understand (such as how the security risks of having a large number of dependencies work)

This video ignores the fact that even many non-malicious packages can have bugs leading to security vulnerabilities, and that many packages have deep dependency graphs that are hard to understand and impossible to audit. The lockfile helps a bit, but doesn't solve the underlying problem. If we had a fully featured, safety-checked standard library, we would need a lot less external packages, and most of these packages would only depend on the standard library anyway.

There is more and more developers these days, that think they know everything from their certificate or boot camp or whatever. It's so easy to find a course and these are the kind of people that says this.

A malicious domain isn't the issue here, it's when there seems to be a culture among many publishers to npm who would install a dependency that itself has a great grandchild dependency instead of writing 30 more lines of code. THAT MENTALITY scales to be a security issue.

i think you completely misunderstood the concern people have. you rely on the npm service and source not getting compromised or having any zero day exploit in it. and then you rely on that being the case for every single packet and dependencies you use. not to forget that the bundler itself is also something that can be compromised. and i think the point is not that this is by default bad or dangerous, but to even notice that it CAN be a problem or risk to blindly trust in stuff like this. and btw the "^" is not there for fun, it has a function and drives your point totally absurd. also the take on dev dependencies never being included in the output is simply wrong.

No, it wasn't about being scared about hosting all those files yourself, that wasn't it at all, and if you think that, you've already failed. The idea of hosting the JS files on a CDN was for many reasons, 1 was for speed, hosting them on a CDN meant that the JS files would be closer to your users, so if your server is in Sydney, Australia, and you have users in say London, they wouldn't have to wait for the files to be pulled from your Sydney server, they would come from the closest CDN node to them. The next reason was for caching purposes, it's possible if you are hosting the file on your own server, it's possible the browser won't use a pre-cached version that it pulled when you visited a completely different site. If it is on the CDN, it's precached based on the CDN, so if you visit 50 different sites, and they all use the same JS from the same CDN, then they only had to download that JS file once, every other site would be using the locally cached version.

the concern is supply chain attacks

Remember, you wouldn't be here if everything was safe.

Trying to chase down peer-deps, warnings when you have no idea what any of it is, only to find allot of the warnings don't matter (for static) is absolute hell.

yeah package management is a hard issue to handle, same happens in python relevant xkcd 1987

This was a fun review. lol I think the issues would still be misspelling a package when installing it. I bet those can still be a bad time when they are hijacked.

"I still don't know what is used for" imagine admitting that you don't know this very basic thing about your development tooling

The npm audit exists solely because of the severity of the issue... and it barely works as we've seen in recent cases. Unfortunately, if a language has a package manager, like npm or cargo, then you'll inevitably have dumb people making projects with huge depdency graphs and sketchy dependencies. Rust's cargo package mamager is in need of an audit feature too because of the same exact problem

How many dependencies does it take to create a button?

Not knowing what package lock is for is unbelievably sad. I don’t write js at all and I know what it’s for.

Hello Theo, the fact that we are using the "^" before any version could actually lead to issue with "yarn" for instance if you are not aware of the --frozen-lockfile. It's not an issue with npm which used this behavior by default on CI, but if we are not very careful and doesn't know our package manager very well, this could lead to unsafe version being deployed without our consent, doesn't it ?

CDN has always always seemed so stupid to me on anything beyond a MySpace page or whatever. HILARIOUS that a site the size of Hulu would blindly load JS to their users from somewhere they do not control. The difference with NPM (which I am sure Theo will say) is that it isn't blind. You can scan you dependencies, you can version lock the code at a place where you know it is safe, and while the code comes from someone else's place, you then put it on your actual server's filesystem so you know it won't change into something malicious. There are definitely issues and concerns with going library-crazy in Javascript-land, but my god people who think that NPM has the same problem as the Polyfill exploit are pretty dumb, tbh.

People just need to accept that pretty much everything in software development is "unsafe" Depedencies have ruined software development. I rely on as little dependencies as possible in my test application and keep them entirely statically linked. I think there are at most maybe 10 dependencies in the test application I maintain.

The problem with NPM really is the granularity and no code quality checks. These two things go together: I you have a dependency graph with 100s of nodes you're never going to check every dependency. You're also not updating individual packages at that point, and just "update all", without any review of source code, or even change logs. The larger the dependency graph the less time you can realistically spend per dependency. I'd love a better NPM, with a staff that actually reviews committed packages for basic style and safety, but that would mean that every idiot suddenly couldn't publish on NPM, which would mean you lost 99% of the JS dev community. This is of course not just a problem with NPM, all the open repositories for code have that same problem, e.g. pip etc., but JS devs seem to depend on a great number of very small packages of unknown quality a lot. Maybe it's the education/documentation we provide with JS: It's always easier to use a library or external dependency, and so a lot of tutorials etc. always use lots of libraries(And for the time-spent in the tutorial that's a good thing; JS devs just don't realize you don't need jquery to get an element by ID).

I've been saying this for a bit. Having your tooling and your frontend code, both written in the same language and installed from the same place, can definitely be confusing.

Excellent! Thank you.

good explanation but you didn't seem to understand the argument