If your captcha takes 20 minutes to complete, your users will leave.

Because obviously there's no middle ground between 0 and 20min…

If your user need to wait for 1s, then it will probably not affect traffic top much (especially if you do it concurrently to loading your bloated webpage that already take a few seconds to load), but you've effectively lowered your spammer's throughput to 1qps (if you're using a memory-hard PoW scheme using Argon-2 or equivalent, the attacker cannot really speed things up by using a beeffier machine, by design of the cryptographic protocol)

Something that takes less than 20 minutes on grandma's old laptop will take less than a second on a powerful server.

Not with memory-hard schemes, that's the point of it actually! And that's because you don't have a 1200x increase in memory latency between your grandma's laptop and a server, you barely have an order of magnitude in the most extreme scenario.

Meanwhile a spammer with the same machine will leave 72 messages every day that you will have to clean up for however long you keep the site up.

And, through the magic of cloud computing, they can multiply 72 by an arbitrarily large number without increasing their cost per captcha.

I get the impression this is mostly only useful against ddos attacks. They do start ramping up pow cost at 5000 requests per second.

The whole point of this system is that the difficulty is set to automatically increase when it detects it's under attack. That's why expected traffic and likely failure traffic are configurable options - when the server is experiencing a higher than normal load the difficulty is ramped up to dissuade those attacks. Yes, when this is happening a real user will also have a slower experience, but they would anyway if the server was being kept busy by the DDoS.

Because spammers use a single piece of hardware matching the exact spec of that low powered devices.

And then spammers will also leave. Problem solved, one way or another.