Every wrong option on a good test question is a trap with a purpose
Take 1/2 + 1/3. A student who adds the tops and adds the bottoms writes 2/5. Put 2/5 among the options, and the student who picks it has just told you exactly what they misunderstand.
That's what a well-built wrong option (a distractor) does: it turns a wrong answer into a diagnosis. But building these traps is slow, expert work, and checking them is slower. A distractor nobody falls for is dead weight. A question where confused students still find the right answer can't tell understanding apart from luck. Today the only way to find out is to put the question in front of thousands of real students and wait for the data.
Rehearse the question with simulated students first
Tell a language model "You are a student who believes [some misconception]", show it the question, and see which option it picks. Do that for many misconceptions and you get a preview of how a classroom would answer, before a single real student sees the question.
That only works if a model can genuinely hold a misconception on command: make that specific mistake, rather than quietly answering correctly or picking any old wrong answer. This first phase tests exactly that. If it holds up, the second phase is an audit tool built on top of it (see where this goes next).
Real questions, expert-labelled mistakes, three ways of asking
We used the Eedi dataset from Kaggle's Mining Misconceptions in Mathematics competition: real maths questions where education experts labelled each wrong option with the misconception that produces it. After setting aside questions that need a picture, that leaves question + wrong option + misconception pairs from questions, covering different misconceptions.
For every pair we asked each model the same question three ways:
Success is picking the option that misconception leads to. We call that a hit.
Same question, unrelated belief. This tells us how tempting the option is anyway.
Can the model solve it at all? This is the knowledge it has to set aside.
Each persona was asked with two wordings: student ("You are a student who believes…") and teacher ("A student believes…; predict their answer"). The options were reshuffled for every call, with the target spread evenly over A–D. Every model answered with a single letter. Every call was cached and every random choice seeded, so each number here can be reproduced exactly.
The Eedi questions can't be published, so here are two of our own, sent to all nine models with exactly the same prompts. Click a wrong option to give every model that misconception, and see what each one picked.
Nine models, from pocket-change to top of the range
To compare like with like, the charts below use the pairs (from questions) that all nine models answered, drawn at random. The five small models also ran on all pairs; those results are in the deep dive and tell the same story.
Yes, and how well depends on the model and whether it gets to think
Hollow dot: how often the option gets picked by a student with an unrelated belief. Filled dot: with the matching belief. The bar behind the filled dot is the 95% range. The gap between the dots is the persona doing its job. Hover any row.
A useful fake student makes its mistake when the misconception applies, and still gets the question right when it doesn't. Up: picks the matching trap. Right: answers correctly when its misconception is unrelated. The arrow is GPT-5.4 with its thinking switched on.
The cost to audit 1,000 questions with a panel of personas each, at each model's measured price per call (log scale), against how often it hits the matching trap.
Deep dive: the same results in more detail
Lift (matched minus control hit rate) with its 95% range, for the five small models on the shared 250 pairs (hollow) and on all pairs (filled).
Lift by topic for the five small models on all pairs, for the ten largest topics. Darker blue means a stronger effect. Hover for hit rates.
For misconceptions tagged on 3+ questions: on what share of their questions does the persona hit? A tool needs most misconceptions on the right-hand side. Small models, full run.
Shared pairs, current wording. Ranges are 95% bootstrap intervals over pairs.
From "can it?" to a tool that checks questions before students see them
Phase 1 says a well-chosen model can hold a misconception reliably enough to be useful. Phase 2 turns that into an audit tool for any multiple-choice maths question:
Pick the panel
Find the misconceptions most relevant to the question, from Eedi's catalogue of .
Run the panel
Each persona answers using the setup that won phase 1: a thinking model with the student wording.
Write the report card
Dead distractor: no persona picks it. Leaky key: confused personas still find the right answer. Gap: a common misconception with no option to catch it, plus a suggested replacement option.
Prove it works
On 100 held-out questions, swap a real distractor for a junk one and check the tool flags it.
Who would use it
Assessment publishers QA-ing a whole item bank overnight instead of waiting for field-test data. Question writers getting distractors aimed at specific misconceptions as they draft. Learning platforms turning each wrong answer into "this student probably believes X" and serving the right follow-up.
What we'd still need
A check against real student answer rates: the models reproduce the expert-labelled mistake, but we don't yet know whether they're as tempted as real children are. Also a licensing answer: Eedi's data is non-commercial, so a product would need permission or its own question bank.
Can a model un-know something on request?
Ask a model that's good at maths to answer as a student who believes 1/2 + 1/3 = 2/5. It knows the answer is 5/6. Can it set that aside and be wrong in exactly the way it was told, or does what it knows keep leaking through?
Whenever a "confused" model misses the intended mistake, the answer lands in one of two places, and they mean very different things:
The intended mistake
It picked exactly the option the misconception leads to.
The correct answer
It couldn't let go of what it knows and answered correctly anyway.
A different wrong option
It did let go of the right answer, but couldn't follow the faulty rule to the right wrong answer.
Same experiment, a different lens
The data is the experiment described on the edtech tab: real Eedi maths questions, each wrong option labelled with its misconception, and every model asked with a matching persona, an unrelated (control) persona, and no persona at all. Here we look at where each answer lands, not just whether it hit, and at what happens when the misconception doesn't apply to the question at all.
One comparison matters most: GPT-5.4 with thinking off and on. It's the same model with the same prompt word for word, and only one setting changes: whether it's allowed to reason privately before answering. Its answer is still a single letter, and we never see the reasoning, only how long it was (about 100 tokens on average).
Knowledge leaks out of weak models; strong models stumble on the mistake itself
Matched-persona answers on the shared pairs. On the right is how often the same model gets these questions right with no persona: the knowledge it was asked to set aside.
The same GPT-5.4 answers to the same items, thinking off (left) and on (right), both wordings combined. Each ribbon is a group of answers moving from one outcome to another. Hover a ribbon.
With the matching misconception
With an unrelated misconception
Gray: right answers with no persona. Orange: right answers as a student whose misconception has nothing to do with the question. Ideally the two dots sit together: an unrelated misconception shouldn't make you worse at this question.
A side note on reading instructions literally
We tried switching on thinking for Claude Sonnet 4.6 too. It did think, but its entire thinking was the answer letter: 7 tokens, every time. Our prompt ends with "Reply with one letter only", and Claude applied that to its private reasoning as well. GPT-5.4 read the same words as a rule about the final answer, and thought for about 100 tokens first. How much a model thinks depends on how it reads the instructions, not just on whether thinking is switched on.
Deep dive: the same results in more detail
Of the misses only: what share went to the correct answer? There are three non-target options, so with no pull at all you'd expect about 1 in 3 (dashed line).
All models pooled, grouped by how often control personas picked the same option. On the left are options nobody falls for by accident.
Each shared pair got matched tries (9 models × 2 wordings). How many pairs got the intended mistake (blue) or the correct answer (orange) k times?
Pretending needs room to think, and clear instructions about where to stop pretending
What we can say
- Models can set what they know aside: with a matching persona, right answers collapse.
- The leak is a small-model problem. Strong models fail by making the wrong mistake.
- Private reasoning is the biggest single lever: it moves answers into the intended mistake and out of everywhere else.
- "Imagine a confused student" can turn into "be confused about everything", especially with the teacher wording.
What we'd test next
- Visible working: let the persona write out its (wrong) working. Does it compute the right answer first and then steer away?
- Prompt wording: does dropping "one letter only" let Claude think, and does that close its gap?
- Real students: are the mistakes models find easy to fake also the ones children make most?