- does the deploy commander create the hotfixes or the engineers who authored the commits?
- it seems that the deployment is fully automated, but engineers still have to be available in case of problems, does that impact productivity?
- "Once we are confident that core functionality is unchanged", is there a particular metric to assert that?
- how long does deployment take currently?
- switching directories doesn't seem like a fully atomic operation yet, isn't there a delay from loading the files and wouldn't that generate 502s from the service? Maybe it's better to create new instances with the new files and then change the router to use those (blue-green)?
With PHP (What slack was using at one point for some of the services. I think everything uses Hack now which may still maintain a similar model). Switching directories can be mostly atomic.
PHP-FPM with opcaching doesn't need to access files once all the opcodes are cached (turn off file modification checks in production). When you move the directory, you will restart the service.
Unless a request hits a file that is rarely used and not cached, you should be not receive any errors moving the directories.
My point is that if there is any downtime for the switch, for example restarting a service, it's not atomic. A small percentage of failed requests can still be high in absolute terms for a company like Slack, so why not using a paradigm [1] where you have atomic switch? And also instant rollback.
- does the deploy commander create the hotfixes or the engineers who authored the commits?
- it seems that the deployment is fully automated, but engineers still have to be available in case of problems, does that impact productivity?
- "Once we are confident that core functionality is unchanged", is there a particular metric to assert that?
- how long does deployment take currently?
- switching directories doesn't seem like a fully atomic operation yet, isn't there a delay from loading the files and wouldn't that generate 502s from the service? Maybe it's better to create new instances with the new files and then change the router to use those (blue-green)?