Shouldn't all your servers have different passwords? How do you manage the passwords in case you're not available when the server reboots?

Have two people know all the passwords. Let each of them encrypt this list of passwords with a password only they individually know. Sync passwords each time someone changes something.

So "yes" – what would you do if you had, say, several hundred (or several thousand) servers?

If I have that many servers then I can assume I have a large budget for security, right?

I would make a password entering automation system. Ensure that that system is dead-simple and secured to death. It must run no other services, firewalled to death even from the intranet, physically secured in a cage, and must be off most of the time. It is only to be turned on when booting a system, and turning it on not only requires a password but also physically walking to the cage, opening it with a physical key, and pressing the "on" button. All target servers must be configured in such a way that they can obtain network access before mounting the encrypted part.

But I'm not a security expert. Maybe I've overlooked something.

Wait, this sounds way harder than the previous solution and entering the password over a remote IPMI console.

It's harder to set up. But when the password entering automation system exists, you just grab the key, go to that system, unlock & boot it, then tell it to enter the passwords for your 1000 servers.

With that many servers you probably have to reboot a few of them pretty often.. You'd have to live in the datacentre :)

I'm not sure you can assume that if you're, say, Instagram (pre-big-round/acquisition). You may well have several hundred, even thousands, of AWS instances with a staff in single digits.

Properly hard problem, as you've pointed out.