I have my TV on which doesn’t bother me but I understand why it’s not ideal for all setups. If I select a song on my iPhone and airplay it to my AppleTV, it does a handover.
If you want to stay within the Apple ecosystem without the TV part, you could use an AVR with airplay built-in. Or get an AirPort Express, which can join a wifi network and become an Airplay client, and connect it via optical (mini toslink) to an AVR. And control it all from a phone or Mac.
At the risk of turning this into "please describe your setup in increasing detail to me," do you use Spotify for anything? I have a Marantz AVR and honestly it's kind of buggy, but I'm trying to connect directly to it from Spotify (mostly because that's just where my listening history is). Perhaps it would do better connecting into an Apple TV and running spotify on there?
I use a Chromecast Ultra for this, works with Spotify, Tidal, Youtube apps. Or, if you don't care about 4K get a regular 3rd gen Chromecast. These are NOT the newest gen Chromecasts with Google TV, these are 'dumb' devices that just receive streams and don't run apps.
Unfortunately as of a few years ago Google TV/Android TV forces ads for useless content in the home screen, taking up bandwidth and slowing load times. The 'dumb' Chromecasts can still talk to the AVR over HDMI-CEC to turn on power, adjust volume, etc.
If you want to stay within the Apple ecosystem without the TV part, you could use an AVR with airplay built-in. Or get an AirPort Express, which can join a wifi network and become an Airplay client, and connect it via optical (mini toslink) to an AVR. And control it all from a phone or Mac.