User: Write “strawberry” one letter at a time, with a space between each letter. Then count how many r’s are in strawberry.
gpt-3.5-turbo: ASSISTANT s t r a w b e r r y
There are 2 r's in strawberry.
After some experimenting, it seems like the actual problem is that many LLMs can’t count.
User: How many r’s are in the following sequence of letters:
S/T/R/A/W/B/E/R/R/Y
gpt-4o-mini: In the sequence S/T/R/A/W/B/E/R/R/Y, there are 2 occurrences of the letter "R."
Oddly, if I change a bunch of the non-R letters, I seem to start getting the right answer.
User: Write “strawberry” one letter at a time, with a space between each letter. Then count how many r’s are in strawberry.
gpt-3.5-turbo: ASSISTANT s t r a w b e r r y
There are 2 r's in strawberry.
After some experimenting, it seems like the actual problem is that many LLMs can’t count.
User: How many r’s are in the following sequence of letters:
S/T/R/A/W/B/E/R/R/Y
gpt-4o-mini: In the sequence S/T/R/A/W/B/E/R/R/Y, there are 2 occurrences of the letter "R."
Oddly, if I change a bunch of the non-R letters, I seem to start getting the right answer.