I've used Uberconference (pro) over fiber for a couple of years, and it's been rock solid. In cases where one caller is bad, the web interface will show you who is killing the call quality and lets you mute them.

Except in rare cases, I find the problems you're describing are usually a problem with the end-user, their equipment, or their network. Cheap phones, bad cell connection, bad voip, etc. An unmuted speakerphone with a noisy background can bring most calls to their knees.