This is a good idea, however one drawback is that Home Assistant sometimes needs "physical access" to devices connected to the computer that it's running on, for example Bluetooth and Zigbee dongles. Of course you can pass through such access to a virtual machine, but it does become somewhat nontrivial.