I worked on the Windows crypto team for a few years. Learned a ton about this area. Most of the work, however, is plumbing. Only a tiny fraction of crypto work is actually on algorithms and that’s mostly performance related.
The interesting thing about Crypto performance tuning is that you really have to ensure that no logical path does a different amount/kind of work than another(i.e. no short circuiting). I used to not think much of it until I saw an RSA private key recovered via acoustic analysis of capacitor whine due to a short circuit condition in a function to multiply two large numbers.(this was using a recent release of openssl) To my knowledge no other area of programming really has this pitfall
If this "I saw" has any further public details, I'd absolutely love to learn more, and I'm pretty sure others would as well.
In particular, I'm especially interested in electrical or real-world attacks - such as capacitor whine! - that can be applied a weakened security situations like asymmetric logic/branching. I vaguely recall CPU voltage fuzzing is a thing, I want to go learn more about that at some point.
I couldn't find the specific lecture/demo that I went to, but I found a video by the same guy with a similar presentation elsewhere. Coincidentally he's also one of the researchers who published the original paper on Meltdown
For the crypto itself, yes. These colleagues had specialized at university and studied afterwards. For the plumbing portions, it’s mostly excruciatingly detail oriented bit manipulation combined with standard windows kernel / core development. I also read a LOT of RFCs.