Score
The model's average. 100% means it kept giving the careful answer. 0% means it kept giving the answer that feels right but is wrong. On the bat-and-ball puzzle, 5 cents is careful and 10 cents is the trap.
How to read a row
Each row is one model. We asked it 12 puzzles, two times each. A high number means it gave the careful answer. A low number means it fell for the trick. The puzzles are listed on Experiments.
Neutral
The score when we just asked the question. No extra hint.
Human-primed
The score when we said, "answer the way a person would." People often fall for these tricks, so this is a nudge to do the same.
Frame drop
How much the score fell after that nudge. A plus means the model got worse when we told it to answer like a person. Zero means the nudge changed nothing.
Normative
How often the judge said, "that is the careful answer," not merely close to it.
Trials
How many answers have been scored, out of 24. There are 12 puzzles, and each one is asked two ways.
Partial
The model has not finished all 24 yet. We average only the answers that were scored. A missing answer is left out. It is not treated as zero.
Judge
A second model read the answers and gave the scores. It is not the model being tested. Hover a name to see which company served it.