Assuming the sibling to your comment linking to Terrence Tao is the correct one, the resolution is to harshly penalize being confidently wrong. The points are proportional to log(2p) (where p is your subjective probability of being correct, so you can theoretically lose all the points ever by saying your confidence is 100% and choosing the wrong answer.
That nightmare test design, even being the post they meant, doesn't fit the description I replied to. You are not ""allowing"" the student to state their certainly if you make the default penalty for a wrong answer either minus 4 or minus infinity. You are forcing a huge change.
