I'm struggling to understand what people think is so difficult about all this data science stuff. The maths is very basic, even in "advanced" ml. Nor is it hard to learn backend software engineering for the purposes of 99% of companies.
It's all about epistemology. How do we know what we think we know? How do we come to know things we didn't know before? And how can we trust those conclusions?
Even if the math is basic, it's really, really easy to draw bad conclusions, look at the wrong problems, not realize that your data is more incomplete than you might think, etc etc etc. Guarding against these bad results - figuring out how to actually manufacture new knowledge - is the heart of the problem.
> How do we come to know things we didn't know before? And how can we trust those conclusions?
IMHO, this is the heart of what discerns scientists from engineers. Yes there is plenty of overlap, but to me this is the principle component. In engineering, the correct answer almost always exists. Enough eyeballs on the problem, sufficient double- and triple-checking converges on high confidence.
Scientific problems may not have a right answer, or gaining confidence has diminishing returns, and at some point you decide that's enough sigmas. You can scrutinize in so many ways but there's always blind spots.
(Data) scientists generally have to be way more comfortable with uncertainty. And as you mention, the easiest person to fool is yourself.
99% of companies? Definitely not. The skills needed to do DS in business or healthcare are not very correlated with doing DS for the physical sciences. Which is the whole point of this comment thread, sure you can understand DL, but you also have to have an understanding of the field to know what type of DL to use. For example, in my role, I came with knowledge of machine learning but had to learn complex fluid physics to be able to know what type of DL techniques to apply or develop.
