The problem with this is that you can be tricked into giving access to it. I would not count plunging a display adapter into my computer giving someone "physical access" but in this case it would do.
Very true. It seems like a simple thing to loudly inform the user that a new PCI device was connected. But that would probably only help savvy people--everyone else would just click ok without reading the dialog.
A real solution would be some sort of DMA whitelist provided by the OS driver.