It's legitimate to call it a background in physics, but given the particular level of background and the context of this particular issue, its relevance is indistinguishable from zero.
Has he ever demonstrated any particular insight or understanding of physics or - more importantly - of physics education? As far as I've been able to find, the answer is no. Not that there's anything wrong with that. At worst it just makes him a typical physics major.
One of the things Karpathy is most famous for, perhaps the thing depending on who you ask are his instructional materials on Deep Learning and Neural Networks, of which at least hundreds of thousands have benefitted.
That's far more tangible than whatever "background" it is you're looking for. He's a good teacher. He stands out for that and that's not an easy thing to do.
Of all the things background doesn't mean much in, being a good educator is at the top of the list. Most Educators including those who've been at it for years are mediocre at best.
The people who educate at the highest level (College/University Professors) are not even remotely hired based on ability to educate so this really isn't a surprise.
Genuinely and I mean no offense, your expectations just feel rather comical. People like Sal Khan and Luis von Ahn would be laughed out of the room looking for your "background".
Sure, Sal is an educator now but he quit being a financial analyst to pursue Khan Academy full time.
The real problem here is that you don't believe what Karpathy has in mind is tractable and not that there's some elusive background he needs to have. His background is as good as any to take this on.
I think you've misread this conversation. I was responding to someone who suggested that Karpathy's "background in physics" indicated some insight into whether this venture, particularly as regards physics education, will effectively give guidance by subject matter experts like Feynman.
If they had cited some other background, like his courses on AI, I would have responded differently.
There's nothing just about that especially when the commenter only said he had a background in physics.