Personally, I find that the Matrix.org server is much better for large rooms. Most of the content is already cached on the server by all the other accounts on there, so joining large rooms takes seconds instead of minutes. Modern sync (v3 already, I think?) also helps a huge amount but it needs client and server support.
I've tried to join the main matrix chat room (#matrix:matrix.org) but my poor homeserver just couldn't handle it.
That's true, but most rooms where this is a problem (the huge ones) aren't encrypted. There's no way you're going to cross-verify everyone in a room with 4000 users in it so encryption in such rooms is essentially useless.
I've tried to join the main matrix chat room (#matrix:matrix.org) but my poor homeserver just couldn't handle it.