You're absolutely right. From my non-lawyer understanding that's why the youtube-dl dmca mainly rests on youtube-dl showing illicit use within their code (download of copyright protected material) and not that the tool is theoretically capable. But it's indeed a slippery road.

There are other examples in the real world tho, where the distribution/creation of the tool is already illegal (e.g. certain weapons or explosives), because only reacting after damage is done is infeasible.

How do you reconcile this view with fair use though?

If fair use tells us that it's ok to use parts of copyrighted works for certain purposes, then there must be a legal avenue for obtaining those parts.

Not necessarily. Public domain and fair use don't require distribution to occur. Ex: Photos taken in the 1890s are public domain, and you may know that they exist (having seen a print in a no-photography-allowed museum), but the owner of the only copy of the photo is under no obligation to distribute them.

Cynically, parts of the DMCA look an awful lot like an end run around fair use.

Lets tackle explosives: Where public interest exists for legitimate use, they are allowed: mining, fireworks, and hobby rockets. And there is real risk of grevious bodilly harm associated, even if used properly - we are not even considering terrorism.

There is massive legitimate use for downloading videos, yet the alleged harm is purely monetary.

Lock picking is illegal in some places such as Hungary.