It has nothing to do with sites. You are missing the point. To access USB device with Linux, any software, including browser, should have permission to access certain files in /dev.
You visit a page. It asks for device access. You get a dialog box choosing the device that matches the filter the site wants. You can either choose a device or decline. Site does not see anything other than what you approve.
You're the one missing the forest for the trees. The security risk is not caused by websites, but by the fact that the browser can access your USB devices in the first place.
By giving the browser access to your USB devices, the browser could act as a keylogger even when you're using other applications.
Further, as there's no proper way to sandbox this, you wouldn't just be giving the browser keylogging capabilities, but any native app running under your user.
You could have an elevated service that has separate configuration for which devices the user wants to grant access to, and it could even work as a proxy to disallow "bad" usage patterns. The interface to USB devices doesn't need to be directly with the kernel.
It's true though that it's difficult to ensure only a certain process has access to it, though the default value set to ptrace_scope by e.g. Ubuntu is a step towards helping that.. But in principle the service can know which executable is issuing the request.
All in all this seems quite a big effort for perhaps not that great benefit. In the meanwhile I'll be using Chromium for WebSerial and WebUSB needs.