You're confusing physical access with ownership. If I have physical access to a device, that does not mean I should have the power to install compromised software on it, because it may not be my device at all.

This is a surprisingly common oversight that the hardware freedom crowd keeps ignoring. I don't understand why so many people just implicitly assume that physical access means security should be thrown out the window.