We're assuming in this scenario that your memory is still basically functional, so the math on a sheet of paper won't be replaced suddenly without you noticing.
If you can't even remember things in the medium term that's a level beyond "trapped in a dream" that's much more hopeless.
So, with that assumption, you make a computer solve a problem that's impossible for a human to work through in a practical amount of time. And then you verify it got the right answer. This proves the math wasn't done by your dreaming brain. (At least it proves it to a pretty good certainty, and you can repeat the test.)
Many NP-complete problems are good candidates here for slow solutions and fast verifications.
Yeah I kept thinking about simulation stuff too but GP's method is a good one for being trapped in one's own mind situation.
Also for situations that have the standard dream level of fidelity, you can try basic reality checks like putting your finger through your hand or more conspicuously spinning along your own axis.