I think something similar already exists. See this, for example: https://koe.ai/recast/

Although I don't know if they're using anything similar to what you suggest. Very cool idea, anyway!