bookshelf-benchmark · October 2026
The question

Can an AI chat model read a bookshelf?

We gave Claude, Gemini and ChatGPT the same six photos of real bookshelves and asked each one: which books can you read here? Then we checked every answer against a hand-verified list of 295 books.

Short answer: carefullyWhat they name is almost always there: 84–98% of the books they listed were really on the shelf.
But not thoroughlyThe best setup found 82% of the books. The weakest found under a third. Crowded shelves hurt most.
What we did
  • Photos6 real bookshelf photos, from a tidy shelf to a crowded one. 295 books in the answer key, each checked by a person.
  • Models7 models from Anthropic, Google and OpenAI. With their thinking settings, that is 12 setups, used in the normal chat apps.
  • TaskThe same prompt for every setup: all six photos in one message, “list every book whose title you can read. Do not guess.” See the prompt ↓
  • ScoringTwo numbers per answer: how many of the books it found, and how many of the books it named were really there. Small differences in wording are forgiven. How ↓
The data

Six real shelves, from tidy to chaotic

  • Two easy, two medium, two hard. The number of books varies a lot, from 8 on photo d to 108 on photo e.
  • The answer key is a list of every book a person confirmed in the photo. Books nobody could read with confidence were left out. How it was made ↓
  • Each title counts once per photo, even with several copies on the shelf.
  • The key is frozen. A model naming a book does not add it to the key, even when the book turns out to be real. Caveats ↓
  • Click a photo to open it larger.

Where the answer key came from

Every model is scored against this list, so it had to be right. It was drafted by an AI working slowly, then checked book by book by a person.

STEP 1

Cut each photo into sections

Claude Opus 5.5 at medium effort, running in Claude Code, split every photo into small sections of the full-resolution image, over a hundred in all, and read the spines in each one.

STEP 2

Draft the full list

From those sections it wrote one list per photo: every title it could read, with the author where it could.

STEP 3

A person checks every book

Each book on the list was checked by hand against the photo: there, or not there. Nothing went into the answer key without a person seeing it.

The biggest lesson of the project started here. The same model did far better inside Claude Code at medium effort, working section by section, than in the Claude.ai chat at high effort, which found 77% of the books on the hardest photo. The harness around a model matters, not just the model. More ↓

The work started with 11 photos. Five still had too many readable books left to check and were dropped, leaving 295 books on 6 photos. The list was then frozen.

Results

Careful readers, not thorough ones

Each tile is one setup on one photo. Pick what the tiles show, and which of the two runs. Click any tile to see every book behind it: what was found, what was missed, and what was named but is not in the answer key.

Rows are setups. Columns are photos, easy to hard.
Crowded shelves split the field.On photo e (108 books), Claude setups found 44–77%. GPT found 31–40% and Gemini 7–20%.
Tidy shelves flip it.On the two easy photos, Gemini 3.8 Flash scored highest of all setups, in both runs.
One run is not enough.Claude Opus 5.5 at low effort went from F1 0.72 to 0.87 between the two runs. Same prompt, same photos.
Experiment 2

One photo per chat: Claude found more books. Gemini did not.

The two hard photos again, each sent in a chat of its own instead of with five others. Each row is one setup, on both hard photos together (147 books). The hollow dot is all six photos at once; the filled dot is one photo per chat. Click a row to see the books.

47% → 78%Claude Opus 5.5 at low effort, the biggest jump. Alone, it matched Opus 5.5 at high effort.
57% → 68%All Claude setups together. What they named stayed right (95% → 92%).
20% → 22%All Gemini setups together. Its limit is reading a crowded shelf, not sharing attention.
Cost

What being right costs

Each point is one setup from run 2. Up is more accurate, left is cheaper. The cost axis is logarithmic. The chat apps show no token counts, so cost is an estimate: what the same request would cost through the API at list prices, with a low-to-high whisker.

Show the numbers as a table
Not obvious from the numbers
  • Claude Opus 5.5 is the most accurate, at about $0.17–0.31 for all six photos. The older Opus 5 cost more ($0.45) and scored lower.
  • Extra thinking pays off unevenly. It added 0.01–0.02 F1 for Opus 5.5, Gemini 3.1 Pro and GPT, and 0.08–0.10 for Sonnet 5.5 and Gemini 3.8 Flash.
  • Gemini 3.8 Flash is the cheapest, about 2 cents, and the best on tidy shelves.
  • ChatGPT was cheap but slow. Even in standard mode it ran Python text recognition on the photos in a loop for five minutes. Its Think mode took 17 minutes.
Bottom line. In a single chat, these models are careful but not thorough at reading a bookshelf. Which one to use depends on the shelf: Claude for crowded ones, Gemini Flash for tidy ones. And how a model is used can matter more than which model it is. Model vs harness ↓
That is the result. Everything below is the how.Optional reading · the story, the method, what went wrong, the caveats
The full story

Ten steps from a bookshop photo to a benchmark

This started as a practical annoyance and turned into a small, repeatable experiment. What follows is how it was planned, what was actually done, and what changed along the way.

1 The itch

“What books are on this shelf?”

It began with photos of messy bookshop and bookshelf displays, and an AI chat model asked to name the books in them. It sort of worked. It also misread spines, missed books, and sometimes named books that were not there.

That became a question that can be measured:

Given bookshelf photos where the real books are known, how well do different models, and different ways of using them, identify the books?

Two rules came with it. Keep it small and quick: a prototype and a web page, not a product. And do not try to crown a best model: measure where each approach succeeds and fails.

2 The answer key

Before scoring a model, decide what the right answer is

A score needs a list of the books that are really in each photo. Building it was the slowest part of the project.

Claude Opus 5.5 at medium effort, in Claude Code, cut each photo into small sections, over a hundred in all, and drafted a full list of the books it could read. Then a person reviewed every single book on that list against the photo and kept only the ones really there. The three steps are at the top ↑

It started with 11 photos. Five still had too many readable books left to check. Keeping them would have marked correct answers as wrong, so they were dropped.

295 books on 6 photos, frozen from then on. Unknown authors are left blank. The five dropped photos and their draft labels are kept in the repository’s archive/ folder.

3 One prompt, three chat apps

Same photos, same words, twelve setups

The plan preferred APIs, for control. There was no API access, so every run used the normal chat apps: ChatGPT, Claude.ai and Gemini. That is closer to how people use these tools, but the apps can resize images, add hidden instructions or run tools, and none of that can be seen.

Each setup got all six photos in one message, in order, with this prompt pasted beside them. The reply was saved exactly as given. Broken JSON would have counted as broken.

ModelSettings run
Claude Opus 5.5low and high effort
Claude Sonnet 5.5low and high effort
Claude Opus 5high effort
Claude Sonnet 5high effort
Gemini 3.1 Prostandard and extended thinking
Gemini 3.8 Flashstandard and extended thinking
GPT 5.6 Lunastandard and thinking

The older Claude models ran at high effort only, so any gap to the newer ones comes from the model. Medium effort was skipped: low against high gives the clearest signal.

I have attached 6 photos of bookshelves, named a, b, c, d, e and f (in that order).

For each photo, list every book whose title you can read.

Rules:
- Only include a book if you can actually read its title in that photo. Do not guess, and do not
  add books that are likely to be there but whose titles you cannot read.
- Keep the photos separate. List each book under the photo it appears in.
- If several copies of the same book are visible in a photo, list it once.
- Include the author if you can read it or are confident who wrote the book. Otherwise use null.
- Write titles as they appear on the book, in their original language and script.

Reply with only this JSON and nothing else:
{ "a": {"difficulty": "easy", "books": [{"title": "...", "author": "..."}]}, "b": ..., "f": ... }
4 Scoring that forgives wording

“Attack on Titan Omnibus” is the same book

Models rarely write a title exactly as the answer key does. So titles are compared loosely: case, punctuation, accents, word order, a leading “The” and a missing subtitle are all forgiven. A pair counts as the same book above 85% similarity. Authors are scored separately and never decide a match.

A surprise: the photos got shuffled

In run 1, every Claude reply looked terrible at first, then turned out to be right under the wrong letters. Its list for “a” was really photo d. The Claude app does not show the model the filenames, so it lettered the photos in its own order. The scorer now finds the best letter-to-photo match and corrects it before scoring. Only whole lists move; a single book under the wrong photo still counts against the model.

Model wroteAnswer keyMatch?
Attack on Titan OmnibusAttack on Titan: Omnibus (Volumes 1-2-3)Yes
Disgusting DigestionHorrible Science: Disgusting DigestionYes
Modern Judgements: RacineRacine: Modern JudgementsYes
You CanYou Can WinNo
Harry PotterHarry Potter and the Goblet of FireNo

Every match and non-match is saved, so any number on this page can be checked book by book.

5 Where it went wrong

Four ways to get a bookshelf wrong

Real examples from the replies. Struck through is what the model wrote; after the arrow is the book on the shelf.

28books that no chat found, across all 42
46–52books that every setup found, in each run

Right author, wrong book

The model reads the author, then names their better-known title.

  • The Magic of the Lost Temple→The Magic of the Lost Earrings · 5 runs
  • Behave→Determined · Sapolsky
  • Bit of a Blur→All Cheeses Great & Small · Alex James

Missed by every model

In run 1, 41 of the 295 books were found by none of the 12 setups.

  • Hindi titles · जादू, मेघनाद, डेरे वाले
  • Upside-down or thin spines · Silas Marner, Hunted
  • Partly hidden covers · The Great Gatsby, Wild Fictions

Too vague to count

Part of the title, not enough to say which book.

  • Harry Potter→Harry Potter and the Goblet of Fire
  • Diary of a Wimpy Kid→Diary of a Wimpy Kid: The Long Haul
  • British Art→The Thames and Hudson Encyclopaedia of British Art

Marked wrong, but really there

On the shelf, but left out of the answer key as too hard to confirm.

  • The Secret Life of Bees · named in 6 run-1 replies
  • Jiggy McCue · 3 in run 1, 5 in run 2
  • The Bitcoin Standard · 2 in run 1, 4 in run 2
6 Too many photos at once?

Was the model rationing its answer?

Run 1 had a pattern: on the hard photos, what models named was right, but they named too few. On photo e most setups found under half the books, and every Gemini setup stopped at about 20–27 titles.

So experiment 2 sent the two hard photos one per chat, with the same rules. If the six-photo message was holding models back, they should list more.

It was, for Claude. Every Claude setup found at least as many books alone, and Opus 5.5 at low effort caught up with Opus 5.5 at high effort. That changes the reading of run 1: most of the gap between low and high effort came from the six-photo setup, not the effort setting.

Gemini barely moved. Alone it still listed only 19–25 books on photo e; it just named fewer wrong ones.

Three setups were dropped: Gemini 3.8 Flash with extended thinking, and both GPT setups, which kept failing in their apps. The chart is at the top ↑

7 Run it again

The same twelve chats, four days later

Cost needed timing data the first chats had not kept, so all twelve setups were run again on 4 October. This time a coding agent did the clicking: Claude Code, driving the chat apps in the Brave browser through the Claude in Chrome extension. It opened private chats, chose the model and setting, attached the photos, pasted the prompt and saved the reply exactly as given. It also read each app’s own conversation data, which confirmed the model used and recorded when it started and stopped thinking.

The rerun doubled as a first look at consistency, and the results moved more than expected.

Things only the rerun showed

  • ChatGPT ran Python text recognition on the photos even in standard mode, 11 code calls in one reply.
  • ChatGPT’s Think mode was served by a different model, gpt-5-6-t-mini.
  • Gemini 3.8 Flash with extended thinking failed once (“something went wrong”) and was retried in a new chat.
8 Counting the cost

An estimate, built from what the apps reveal

Chat apps do not report tokens, so each run was priced as if the same request had gone through the API at list prices, as a low, middle and high estimate. The chart is at the top ↑

  • Input: the prompt plus the six photos, by each provider’s published image-token rule. Claude: up to about 4,800 tokens per large photo. Gemini: 1,120 per photo. GPT: one token per 32×32 patch.
  • Output: the saved reply, plus thinking. Thinking is hidden, so it is estimated from time: how long the model thought, times how fast the same run wrote its reply.
  • ChatGPT: its Python loop re-read the photos on every turn, and its hidden reasoning is unknown, so its range is the widest.
  • Hidden system prompts are not counted. Good for comparing orders of magnitude, not cents.
9 The model, or the way it is used?

Same model, very different result

The answer key was drafted by Claude Opus 5.5, the model that scored best here, at medium effort in Claude Code. Working section by section, it listed nearly every readable book. In the Claude.ai chat at high effort, the same model found about 77% of the books on the hardest photo.

The model did not change, and it even had less effort. The way it was used did. The biggest factor is probably resolution: chat apps shrink large photos, and in a shrunk 24-megapixel photo a spine title can be a few pixels tall. The harness around a model can matter more than the model or its settings. This was the biggest lesson of the project.

The planned verification harness (identify, flag the unsure ones, zoom in, re-check) was not built, because it needs API access. This is an observation, not a measured result.

Drafting the answer keyA single chat
WhereClaude Code, medium effortClaude.ai chat, high effort
DetailSmall sections of the full-resolution photoThe whole photo, usually shrunk by the app
PassesOver a hundred sections, one at a timeOne look at six photos
CheckingA person reviewed every book on the listNone
10 Planned, done, and what it means

What was planned against what happened

The plan had a first benchmark, then a harness experiment, then a set of “does the packaging matter?” experiments. The benchmark is done and went further than planned. Most of the packaging experiments are still open.

PlannedStatus
Frozen answer key, 6 photosDone
2–3 models, one standard promptDone 7 models, 12 setups
Scorer: precision, recall, F1, per photoDone
Failure examples and a results pageDone
Use APIs for controlChanged chat apps only
All photos at once vs one per promptDone hard photos
Repeat runs for consistencyPartly 2 runs
Cost per runDone estimated
Direct prompting vs a verification harnessNot done needs API
ZIP vs separate imagesNot done
Image compressionNot done
PDF, Markdown, base64 and other formatsNot done

Read with care

  • Two runs at most. Scores moved by up to 0.15 F1 between runs, so small gaps between setups may be noise.
  • Chat apps, not APIs. Each app may resize photos, add hidden instructions or run tools. A result means “model X in app Y on that date”.
  • “Not in the answer key” is not always wrong. Some real books were left out as too hard to confirm, so precision is slightly understated for everyone.
  • Loose matching can slip. It can pair two similar titles, and a Hindi title written in Latin letters does not match the same title in Devanagari.
  • Six photos. Enough to see patterns, not to rank models in general.

What could come next

  • A zoom harness: cut each hard photo into overlapping full-resolution crops, ask about each, merge the lists. That would measure how much of the gap is resolution alone.
  • More repeat runs, to put error bars on every score.
  • Compression: original against moderately and heavily compressed photos, for accuracy and cost.
  • Packaging: the same photos as a ZIP, a PDF or a document.

Run it yourself

Every photo, prompt, raw reply and score is in the repository. Scoring needs only Python 3, with nothing to install.

python benchmark/evaluate.py         # experiment 1
python benchmark/evaluate_single.py  # experiment 2
python benchmark/estimate_cost.py    # run 2 cost
python benchmark/build_site.py       # this page

Repository  ·  The full story, with more detail  ·  Experiment notes  ·  How scoring works  ·  Every raw reply