I have a CS PhD from a good (but not top 5) program and have published in top venues. I perceived my chances of getting a job at Microsoft Research (MSR) to be low (on par with getting a faculty position at a top place) and didn't feel like going through the unpleasant prospect of coming up with job talks and slides, sending off to universities while I'm at it, writing diversity statements, etc, for fairly low chances at getting something at MSR.
I could have gotten something at a 'lesser' place, but my guess is they'd be even more likely to be disrupted by budget cuts.
Even getting one of these roles though leaves you in a position where the next guy who wants to save money can just axe the whole division and then you're on your ass and due to a general paucity of research roles and high competition, this can be very, very bad.
I went bog-standard industry and the PhD probably didn't help much there. My industry job largely wastes my training and research experience. In retrospect, I was foolish to get a PhD and people choosing to not do so are generally making the right choice.
My understanding from PhDs in the research business at major companies is that once you're in the club, it's a lot easier to get a position at another one.
You'd want to keep your network alive. Publish, be on program committees doing reviews, etc. If the number of jobs is fairly static and you're doing this and your area doesn't collapse, (e.g., the AI winter), your network can probably land you okay. Networks in the academic world is a pretty big deal and it's a small world. Some people like doing this kind of 'service,' I wasn't one of them.
If there are general headwinds (i.e., research spending in general drops, which seems to have been an ongoing trend), there is almost definitionally more people getting cast off than there are actual researcher roles, not including new entrants to the market.
As with all things, the better off you do (very high quality lab, for instance), the more places you have to try to grab onto if you get cast off.
My peers from grad school have gone every which route, industrial labs, academia, more applied research-lite positions, finance, and fairly direct software engineering jobs. A decent chunk that started in academia or at labs have migrated into more standard software engineering roles. Personally, I really miss research, but it is what it is.
I have anecdata that shows that plenty of intelligent, highly motivated, affable people with (hard science) PhDs still struggle to obtain employment of the "club" caliber, even after they're in the "club".
I could have gotten something at a 'lesser' place, but my guess is they'd be even more likely to be disrupted by budget cuts.
Even getting one of these roles though leaves you in a position where the next guy who wants to save money can just axe the whole division and then you're on your ass and due to a general paucity of research roles and high competition, this can be very, very bad.
I went bog-standard industry and the PhD probably didn't help much there. My industry job largely wastes my training and research experience. In retrospect, I was foolish to get a PhD and people choosing to not do so are generally making the right choice.