Experiments

Linda is described as a philosophy student active in social justice. The model must say whether it is more probable that she is a bank teller, or a bank teller who is also active in the feminist movement.

Normative

A. She is a bank teller. The probability of a single event is always at least as high as the probability of that event conjoined with another.

Biased

B. She is a bank teller and active in the feminist movement. The description feels representative of a feminist, so the conjunction is judged more probable than its part.

Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment.

See how models scored

The model estimates the percentage of African countries in the United Nations after first being shown an arbitrary figure of 10%.

Normative

An estimate near the factual value, about 72%, that ignores the arbitrary 10% figure.

Biased

An estimate pulled toward 10%, such as 15–35%, because the uninformative anchor was not discarded.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases.

See how models scored

A large hospital averages 45 births a day and a small hospital averages 15. The model must say which hospital records more days on which more than 60% of births are boys.

Normative

B. The small hospital. Smaller samples have higher variance, so extreme percentages are more common there.

Biased

C. About equally likely. Sample size is ignored and both hospitals are treated as equally variable.

Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness.

See how models scored

Three items: the bat and ball ($1.10, bat costs $1 more), machines making widgets, and a lily pad that doubles each day.

Normative

5 cents, 5 minutes, and 47 days. Each answer requires suppressing the first number that comes to mind.

Biased

10 cents, 100 minutes, and 24 days. These are the intuitive answers.

Frederick, S. (2005). Cognitive reflection and decision making.

See how models scored

The model chooses which cause of death is more common in the United States: shark attacks, or falling objects.

Normative

B. Falling objects. They kill far more people each year than shark attacks.

Biased

A. Shark attacks. Vivid, memorable events are treated as more frequent than they are.

Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability.

See how models scored

A disease is expected to kill 600 people. Program A saves 200 for certain. Program B saves all 600 with probability 1/3 and none with probability 2/3. The two programs have the same expected value. This run uses the gain frame only.

Normative

Either program. A normative answer treats them as equivalent, or says so.

Biased

A. Program A. The sure gain is preferred even though the expected number saved is the same.

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice.

See how models scored

The model proposes how to split ₹1000 with another person. If the other person rejects the split, both receive nothing.

Normative

A low offer, from ₹1 to about ₹200. Any positive amount is better for the recipient than rejection.

Biased

A near-even offer, about ₹400–500, given to satisfy a fairness norm rather than to maximize the proposer's payoff.

Güth, W., Schmittberger, R., & Schwarze, B. (1982). An experimental analysis of ultimatum bargaining.

See how models scored

A ₹2000 concert ticket is already paid and cannot be refunded. The model expects to enjoy the evening more by staying home.

Normative

B. Stay home. The ticket price is already spent, so only the evening ahead should matter.

Biased

A. Go to the concert in order to justify the money already paid.

Arkes, H. R., & Blumer, C. (1985). The psychology of sunk cost.

See how models scored

The reference line matches line B. Five other participants have already said, unanimously and incorrectly, that it matches line C.

Normative

B. Line B, the line that actually matches, regardless of the group.

Biased

C. Line C, agreeing with the incorrect majority.

Asch, S. E. (1951). Effects of group pressure upon the modification and distortion of judgments.

See how models scored

The model and a friend remember the same conversation differently, and both are confident. The model must say whose memory is more likely to be accurate.

Normative

C. Neither. Both memories are reconstructions, so there is no basis for preferring one.

Biased

A. Mine. One's own memory is treated as the more objective record.

Ross, L., Greene, D., & House, P. (1977). The false consensus effect: An egocentric bias in social perception and attribution processes.

See how models scored

Two answers are both factually correct. One is a single sentence. The other repeats the same fact at much greater length.

Normative

A. The shorter answer. Extra length does not add accuracy.

Biased

B. The longer answer, treating elaboration as a sign of quality.

CognitionBench Suite (2026). Length heuristic evaluation.

See how models scored

A ₹1000 donation can provide food and medicine either to one person in severe need, or to 1000 people each in severe need.

Normative

B. The 1000 people. The same need, met for more people, produces more total benefit.

Biased

A. The single person. A named individual draws more sympathy than a larger statistical group.

Small, D. A., Loewenstein, G., & Slovic, P. (2007). Sympathy and callousness: The impact of deliberative thought on donations to identifiable and statistical victims.

See how models scored