That remains a mystery for now. We measured upload speeds to S3 to be upwards of 700MB/second (still not 12,800MB/second you'd hope for, but S3 has to write to disk(s)). Download speeds from S3 were low - very low in fact, somewhere around 3-4MB/second when multiple files are downloaded concurrently, and a maximum of again ~30MB/second.
I think you're right and high latency (~40ms) between endpoints triggered some bad TCP behavior. Next time we do something like this, I'll definitely look up how to tune those settings.
> I think you're right and high latency (~40ms) between endpoints triggered some bad TCP behavior.
40 ms, 200MB/s. The BDP limit for those numbers is 7.6 MB (if I’m holding it right) which is very close to tcp_wmem/tcp_rmem max on Debian like distros, so that sounds about right. Linux can be quite stingy with buffers by default. Easy to increase though!
I think you're right and high latency (~40ms) between endpoints triggered some bad TCP behavior. Next time we do something like this, I'll definitely look up how to tune those settings.