When I prompted GPT-4 with their exact prompt, it gave me the correct answer (for the cat one at least):
SYSTEM: In each example, you will be given some context and a claim,
where the correctness of the claim is affected by some ambiguity in the context. Enumerate two or three interpretations of the context that lead to different judgments about the claim.
Example
Context: {premise}
Claim: {hypothesis}
Given the context alone, is this claim true, false, or inconclusive?
We don’t know, because the context can be interpreted in many
different ways:
1. {disambiguation 1} Then the claim is true.
2. {disambiguation 2} Then the claim is false.
3. {disambiguation 3} Then the claim is inconclusive
USER: P: I’m afraid the cat was hit by a car.
H: The cat was not hit by a car.
ASSISTANT: Given the context alone, is this claim true, false, or inconclusive?
We don't know, because the context can be interpreted in many different ways:
1. The person is stating a fact: The cat was indeed hit by a car. Then the claim is false.
2. The person is expressing a fear or concern, but it hasn't actually happened: They are worried that the cat might have been hit by a car, but they don't have any evidence or confirmation. Then the claim is inconclusive.
3. The person is using figurative language or exaggeration: They might be using the phrase "hit by a car" metaphorically to describe a difficult situation the cat is in, but the cat was not literally hit by a car. Then the claim is true.
SYSTEM: In each example, you will be given some context and a claim, where the correctness of the claim is affected by some ambiguity in the context. Enumerate two or three interpretations of the context that lead to different judgments about the claim.
Example
Context: {premise}
Claim: {hypothesis}
Given the context alone, is this claim true, false, or inconclusive?
We don’t know, because the context can be interpreted in many different ways:
1. {disambiguation 1} Then the claim is true.
2. {disambiguation 2} Then the claim is false.
3. {disambiguation 3} Then the claim is inconclusive
USER: P: I’m afraid the cat was hit by a car. H: The cat was not hit by a car.
ASSISTANT: Given the context alone, is this claim true, false, or inconclusive?
We don't know, because the context can be interpreted in many different ways:
1. The person is stating a fact: The cat was indeed hit by a car. Then the claim is false.
2. The person is expressing a fear or concern, but it hasn't actually happened: They are worried that the cat might have been hit by a car, but they don't have any evidence or confirmation. Then the claim is inconclusive.
3. The person is using figurative language or exaggeration: They might be using the phrase "hit by a car" metaphorically to describe a difficult situation the cat is in, but the cat was not literally hit by a car. Then the claim is true.