Given that people probably don't arrange their life around the ability to safeguard physical assets, I think you've just given several good arguments against self-custody:
Many people don't own a home. Not all of those that do want to expose themselves to the liability of storing valuables there. The safety situation between homes even within the US varies greatly, and even more so internationally. Banks with safety deposit boxes aren't ubiquitous.
If it works for you, great! I'm just saying that you might be in the minority here.
If your threat model includes someone inside your home with power tools and you can’t trust a bank… you might want to consider moving.