I have a stereo pair of Homepods and they work perfectly. Maybe the wifi channel of your AP is congested? Surprisingly, the Homepod also works when there is no wifi available.
It still boggles the mind why there is no wired input though.
On my home network, the HomePod Mini's demand to use both IPv4 and IPv6 is what I'm thinking is contributing to the unreliability of my stereo pair. My ISP doesn't provide me an IPv6 address, so my home router and pihole both seem to struggle with the HomePod Minis. My original HomePod doesn't have this issue.
What I'd really love is if my HomePod Minis stopped assuming the primary HomeKit Hub role and instead let my Apple TV connected via Ethernet handle all of the requests. There's no way to select a device to be the primary hub as far as I can tell.
It still boggles the mind why there is no wired input though.