You seem to know a lot about this area. I do not but I've heard explaining what deep learning models do is a black-box? If you work in a "misson critical" you'd have to explain all the math behind the model. Let's say in healthcare, finance, aviation, etc.

Also, the "big math brain"'s you're talking about probably read all the books your shutting down. I'd say their big math brains are the reason we have LLMs today.