With AMD cards it's relatively easy if you're willing to install two cards and have one of them just sit there doing nothing when you're not using the VM. It's also possible to use just one card and detach it from the host system, pass it to the VM, and then reattach back to the host system when you're done playing, although I spent multiple days on this and never got it working. YMMV, it was 2-3 years ago, the driver support may have improved.
With nvidia it ranges from difficult to impossible.
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
With AMD cards it's relatively easy if you're willing to install two cards and have one of them just sit there doing nothing when you're not using the VM. It's also possible to use just one card and detach it from the host system, pass it to the VM, and then reattach back to the host system when you're done playing, although I spent multiple days on this and never got it working. YMMV, it was 2-3 years ago, the driver support may have improved.
With nvidia it ranges from difficult to impossible.