It's nearly impossible to predict recovery without understanding the system. You would probably need to know how ssh is configured, how secrets are managed, and how files are distributed, both before and after puppet.
Circular dependencies can absolutely wreck you. For example, puppet could configure sudoers, and without puppet config being applied people who would normally expect access might not have it. So now you have to find a privileged ssh key for un-configured machines.
I would be surprised if twitter did not have a physical vault with a USB drive with a root SSH key on it. With that you can do just about everything.
I would be most terrified of machine churn. Auto-remediation systems or elastic capacity systems can result in lost capacity that can't come back until the configuration problem is resolved.
Circular dependencies can absolutely wreck you. For example, puppet could configure sudoers, and without puppet config being applied people who would normally expect access might not have it. So now you have to find a privileged ssh key for un-configured machines.
I would be surprised if twitter did not have a physical vault with a USB drive with a root SSH key on it. With that you can do just about everything.
I would be most terrified of machine churn. Auto-remediation systems or elastic capacity systems can result in lost capacity that can't come back until the configuration problem is resolved.