Sure. Let's even leave aside the whole corpus of data you need to make a even remotely useful voice assistant.
But if you think there will be people willing to self host and maintain a whole voice assistant service, you're off your rocker.
It's just like saying "Why doesn't everyone just use Tor and PGP?" Because it's an enormous pain in the ass and really only accessible to technical users.
I think you shouldn’t rush to judge the utility; consider that other people might find uses for a voice assistant that doesn’t require “a vast corpus of data” to function.
But moreover, if you need to call me “off my rocker,” you’re not engaging in good faith.