Depfu now supports pnpm

tldr: Depfu launched a beta version of pnpm support that should work for most people using pnpm. If you find any issues, please let us know.

We’ve had requests to support pnpm for a long time and we finally got around to implementing it. Deciding to support a new package manager, even if adjacent to existing implementations is always tricky. On one hand, you would think that it instantly grows our audience, at least when it is a popular choice. On the other hand, every package manager is a new breeding ground for bugs and their gentler sisters, edge cases.

For us, one deciding factor is the adoption outside of open source projects. While having Depfu available for free for open source projects is great marketing and also a service we love to provide, it is only from private projects that we generate revenue that allows us to provide the free service.

For pnpm, to us it looks like it now has enough adoption that we can justify supporting it.

Pnpm certainly is an interesting package manager. It makes some technological choices that make it very enjoyable to use as a developer. Most people would probably praise its speed and it certainly is impressive, especially in contrast to npm itself.

Supporting pnpm on Depfu was not without its challenges, though.

A wild yaml file appears

The most interesting one from a tech perspective is the lockfile format. It is saved as YAML, which is somewhat standardized, but there are lots of different parsers and generators out there and they all treat YAML slightly different. Nevertheless, I decided that I didn’t want to build a process that would need to run the actual pnpm yaml parser, but instead just use Ruby’s psych library.

Unfortunately, in the mission to (I think) both reduce file size and make git merges as easy as possible, pnpm made some, let’s say, edgy choices on how to generate their lockfile. I’m not complaining too much here, because you would normally assume that pnpm is the sole distributor and consumer of these files. Unless you aren’t. Turns out that you actually need a very modern version of libyaml, the underlying C-library of psych, to be able to parse all constructs that pnpm uses to save space. This leads to some interesting challenges in making sure psych can use the latest libyaml version on all platforms Depfu code is run on. We’ve been maintaining our own docker base image for quite some time now and this was a good reminder why.

When caches don’t work

The next challenge sounds a bit counter intuitive but one of the things that makes it quite problematic to run pnpm is its aggressive caching strategy using the content store. This is the core thing that makes pnpm fast, because it only creates symlinks instead of downloading or copying files.

The way that pnpm has implemented this means that it is really fast in day to day local use, but actually quite slow when used with a cold cache or rather an empty content store. And for safety reasons, we cannot keep the content store between runs. The scenario of leaking a private package of a customer is rather theoretical but it’s one of the things you want to prevent to even happen theoretically.

Another by-product of this aggressive caching strategy is that pnpm never seems to use the npm package API to look up anything else but the download URLs, so it actually downloads all packages while resolving the dependency tree. There is a command line switch --lockfile-only that, in theory, should only operate on the lockfile (and npm, for example, only downloads packages for which the API doesn’t work like git dependencies), but in practice it only doesn’t link the packages into the node_modules folder.

These two issues combined mean that unfortunately pnpm is a bit of a pain to operate as it is not only slow but also needs quite a bit of system resources.

Technically, these are valid tradeoffs to be made. For a package manager, providing a better developer experience will always be more important than optimizing for this one weird use case. There is, though, something to be said for tools that can do both and in that respect, pnpm has a long ways to go at this point.

In the upcoming weeks, while we iron out the last kinks in the pnpm implementation we’ll figure out how to properly feed this back into the pnpm development process.

Dependency patching. The good, the bad and the ugly.

I need to mention one thing in the end that seems to be very popular with the pnpm crowd that I find quite troubling and that is built in dependency patching.

First introduced in yarn 2, I think, but also available in other ecosystems (for example, composer has a plugin for that), pnpm also supports a workflow for directly patching dependencies.

First of all: I know why this feature exists. It’s often not possible to quickly get patches into a package that is broken for you. Been there, done that.

But I think yarn 2/3 and pnpm make it way too easy to quickly patch packages in your dependency tree. And a quick scan over a couple of pnpm based repositories seem to support that point. If you have patches in maybe half of your direct dependencies, it points to something being extremely wrong.

The way I see it (And I realize that this is not necessarily a popular opinion), being able to patch dependencies locally makes it really easy to skip the step of at least trying to get it applied upstream. Providing a patch to an open source project isn’t always fun, it’s not even always possible and it takes time your bosses probably don’t want you to spend on that. Still.

Adding to that, most patches I’ve seen applied in the wild are untested quick hacks. They never actually ran through a CI suite (Apart from maybe being accidentally being tested in a test of your application) and they are stored in a format that is really kind of hard to reason about.

Git dependencies do exist

I fail to see why the alternative to fork the upstream project and then use a git dependency is such a big issue. It gives you a quick way to provide a patch to the upstream project. It allows you to add to their test suite. Your changes exist in the form of proper git commits instead of a patch file in a directory.

Coming back to Depfu and the job we’re trying to do, it doesn’t matter much which solution you’re using, but we’ve built support for git dependencies ages ago.

For patches, our only real solution is to ignore the patched dependencies, which was something we had to add as a new feature to Depfu. The reason is, of course, that patches will often not apply cleanly on a new version of a dependency.

What we’re currently missing is a possibility to show these cases in our dashboard - We’ll add that later. For now, know that if you use pnpm patch, you’re opting out of dependency updates for that dependency until you’re removing the patch.

What works, what doesn’t

In terms of features of pnpm we’re currently not supporting:

• We don’t have explicit support for branch lockfiles (And we will ignore any lockfiles not named pnpm-lock.yaml).
• We also don’t support plug-and-play in that the resulting modules file will not be part of the pull request.
• Last but not least we currently also don’t support .pnpmfile.cjs and probably never will as we can’t really control what’s going on in there.

Building support for pnpm was a bit more involved than we thought and we are interested to see what weird edge cases you all can come up with to keep us busy 😊.