I happen to have it on good authority that there is an endpoint lurking somewhere in our API that has a p99 latency of almost 3 full hours per request. We clearly don't respect our users time. Or our own. Or basic software engineering practices. Sigh.
Yeah client's will timeout for sure after a few minutes. The API doesn't know that though... it'll just sit there wasting an ungodly amount of resource and causing tonnes of noisy neighbor problems. You look in the APM traces and see one with 8 million spans that enumerates the whole database. Just silly. It's easy to be a 10x engineer in an environment where there is a lot of 0.1x engineering :(
