I've had to implement this exact logic at work, because we have to talk to devices using modbus tcp, where a lot of devices only supports having one active request per connection at a time. One device we talk to only supports having 5 active connections to it.
I've dealt with low-limit APIs, and I've always considered the better approach to the problem is to create a proxy rather than trust the clients to manage, limit and coordinate themselves.
Then we send all traffic to proxy.service instead of canon.service, and implement queuing, rate limiting, caching, etc on the proxy.
This shields the weak part of the system from the clients.
I guess I hadn't considered the existence of generic TCP proxies
And I'd probably have concerns around how much latency it would introduce in an environment where we have some requirements on the rate of data collection.
That said our service Does act as a kind of proxy for a few protocols.
