At at previous job, we made an HSM. We spent a lot of time considering manufacturing and evil maid attacks. There are solutions.

The biggest important leap of faith is that the manufacturer must be able to lock themselves out of the device. This involves physical tamper resistance and people processes. And lots of key management.