I had not even 3 users... I was running a Java one, looks like Prosody is worth it's salt. Does it support E2E and other features that Conversations supports? :)
Edit:
I was running it on a DigitalOcean box with only 512MB of ram at the time, which I found a bit silly that I would come back and the server was down, but it was a different XMPP server than Prosody.
You probably used Openfire, known to be a memory hog and a bit unstable.
Simply go for prosody and you should be fine (ejabberd should also be OK). Some newer XEPs which you may want to have with Conversations are not supported OOTB in prosody (yet) but there are plugins distributed separately [1][2].
Yes that was the culprit! OpenFire sounds like it... I wanted something simple to setup, without too many terminal incantations. Thanks for the information, it's invaluable.
E2E does not need support from the server. Regarding Conversations, I'm using it with my server and it seems to be working fine, although I have not tested every single XEP supported by Conversations.
Edit:
I was running it on a DigitalOcean box with only 512MB of ram at the time, which I found a bit silly that I would come back and the server was down, but it was a different XMPP server than Prosody.