You are absolutely right, but this is quite an advanced level of mathematical maturity, I would say Ph.D. level.
Someone who is not pursuing a math education can still get to this level, especially if they want to get a deep understanding of ML. You simply try to implement some of the ML algorithms using the math that you learned. If your math is wrong, the ML that you implement does not work. Debugging that is difficult, but once you get good at that type of debugging, you acquire the skills to self-check your math.
Until then, just successfully getting the answers to the Khan Academy exercises should do the trick.
Someone who is not pursuing a math education can still get to this level, especially if they want to get a deep understanding of ML. You simply try to implement some of the ML algorithms using the math that you learned. If your math is wrong, the ML that you implement does not work. Debugging that is difficult, but once you get good at that type of debugging, you acquire the skills to self-check your math.
Until then, just successfully getting the answers to the Khan Academy exercises should do the trick.