Quantum mechanics: frustrating formal mathematicians for over a century.

You'd have to exercise caution when using (quasi-)probabilities less than zero or greater than one, because many of the classical properties of probability theory depend upon those conditions.