Do you have a better solution in mind? I'm actually wondering because I've thought of workarounds but they'd all be hacks like the ones in the handbook. I'd imagine other node developers are in the same boat (not due to stubbornness) otherwise we would've seen something by now.
The obvious thing is what most other environments do: there's a system default search path, you can override that using an environment var, and then once your runtime is up, you can further override it via an API at runtime. E.g. with ruby there is an array called $: which is the current search path in array-of-strings form, and manipulating that array affects the module loading behavior. This provides a lot of flexibility in how apps find their modules. Node provides basically none.
Dictating that the only way to control dependency-loading is by manipulating the filesystem is really braindead.
Unfortunately in that first link, isaacs is on record as saying we should just check our code into a node_modules directory. "Problem solved." This is exactly the kind of stubbornness I'm talking about.
All the pain and exhaustion is a sign you're fighting your way uphill instead of downhill. Node works brilliantly if you publish lots of tiny modules in public, @substack style.
Unfortunately, some of us need to live uphill.
I find the tiny module style works pretty well for me, but I need private code. I haven't yet found a private registry solution I like. NPM's lack of support for any "get this module from this registry" directive in package.json forces them to proxy the main registry. They're not good at it. I'm still declaring my dependencies by tarball URL, losing semantic version specs, e.g. "most recent compatible with 4.x".
It strikes me as unlikely now to change in any way other than "here, use my paid-for private registry", because shareholders. That makes me unhappy, but not unhappy enough to ditch the public Node ecosystem and go back to what I was using before. It had its own uphill, and living there was harder.
One thing that amazes me is the arrogance with which people who like this scheme can make pronouncements about right and wrong ways to organize code.
From the most popular answer on the stackoverflow link: "If you find yourself loading common files from the root of your project (perhaps because they are common utility functions), then that is a big clue that it's time to make a package."
I'm sorry, but no, that is not a big clue that it's time to make a package. Every project I've ever worked on contained internal modules which were nicely organized and isolated, and were consumed by the rest of the project as black box "libraries" for good design's sake, but breaking them out into separately installable modules would have been sheer, unproductive bureaucracy. Oops, you made your code modular! Get ready to start managing packages!
It's as if npm no longer wants to be merely a package manager, but also have a say in how your project internals are laid out as well. Perhaps it's a self-perpetuation strategy.
I think you might be attributing stronger emotional intent than is present. They disagree about whether it's worth fundamentally changing npm's search method to avoid some ../.. paths in your argument to require. require aside, you can express yourself through your code however you like.
Basically, what would be your ideal solution?