Six real shelves, from tidy to chaotic
- Two easy, two medium, two hard. The number of books varies a lot, from 8 on photo d to 108 on photo e.
- The answer key is a list of every book a person confirmed in the photo. Books nobody could read with confidence were left out. How it was made ↓
- Each title counts once per photo, even with several copies on the shelf.
- The key is frozen. A model naming a book does not add it to the key, even when the book turns out to be real. Caveats ↓
- Click a photo to open it larger.
Where the answer key came from
Every model is scored against this list, so it had to be right. It was drafted by an AI working slowly, then checked book by book by a person.
Cut each photo into sections
Claude Opus 5.5 at medium effort, running in Claude Code, split every photo into small sections of the full-resolution image, over a hundred in all, and read the spines in each one.
Draft the full list
From those sections it wrote one list per photo: every title it could read, with the author where it could.
A person checks every book
Each book on the list was checked by hand against the photo: there, or not there. Nothing went into the answer key without a person seeing it.
The work started with 11 photos. Five still had too many readable books left to check and were dropped, leaving 295 books on 6 photos. The list was then frozen.
Careful readers, not thorough ones
Each tile is one setup on one photo. Pick what the tiles show, and which of the two runs. Click any tile to see every book behind it: what was found, what was missed, and what was named but is not in the answer key.
One photo per chat: Claude found more books. Gemini did not.
The two hard photos again, each sent in a chat of its own instead of with five others. Each row is one setup, on both hard photos together (147 books). The hollow dot is all six photos at once; the filled dot is one photo per chat. Click a row to see the books.
What being right costs
Each point is one setup from run 2. Up is more accurate, left is cheaper. The cost axis is logarithmic. The chat apps show no token counts, so cost is an estimate: what the same request would cost through the API at list prices, with a low-to-high whisker.
Show the numbers as a table
- Claude Opus 5.5 is the most accurate, at about $0.17–0.31 for all six photos. The older Opus 5 cost more ($0.45) and scored lower.
- Extra thinking pays off unevenly. It added 0.01–0.02 F1 for Opus 5.5, Gemini 3.1 Pro and GPT, and 0.08–0.10 for Sonnet 5.5 and Gemini 3.8 Flash.
- Gemini 3.8 Flash is the cheapest, about 2 cents, and the best on tidy shelves.
- ChatGPT was cheap but slow. Even in standard mode it ran Python text recognition on the photos in a loop for five minutes. Its Think mode took 17 minutes.
Ten steps from a bookshop photo to a benchmark
This started as a practical annoyance and turned into a small, repeatable experiment. What follows is how it was planned, what was actually done, and what changed along the way.
“What books are on this shelf?”
It began with photos of messy bookshop and bookshelf displays, and an AI chat model asked to name the books in them. It sort of worked. It also misread spines, missed books, and sometimes named books that were not there.
That became a question that can be measured:
Given bookshelf photos where the real books are known, how well do different models, and different ways of using them, identify the books?
Two rules came with it. Keep it small and quick: a prototype and a web page, not a product. And do not try to crown a best model: measure where each approach succeeds and fails.
Before scoring a model, decide what the right answer is
A score needs a list of the books that are really in each photo. Building it was the slowest part of the project.
Claude Opus 5.5 at medium effort, in Claude Code, cut each photo into small sections, over a hundred in all, and drafted a full list of the books it could read. Then a person reviewed every single book on that list against the photo and kept only the ones really there. The three steps are at the top ↑
It started with 11 photos. Five still had too many readable books left to check. Keeping them would have marked correct answers as wrong, so they were dropped.
295 books on 6 photos, frozen from then on. Unknown authors are left blank. The five dropped photos and their draft labels are kept in the repository’s archive/ folder.
Same photos, same words, twelve setups
The plan preferred APIs, for control. There was no API access, so every run used the normal chat apps: ChatGPT, Claude.ai and Gemini. That is closer to how people use these tools, but the apps can resize images, add hidden instructions or run tools, and none of that can be seen.
Each setup got all six photos in one message, in order, with this prompt pasted beside them. The reply was saved exactly as given. Broken JSON would have counted as broken.
| Model | Settings run |
|---|---|
| Claude Opus 5.5 | low and high effort |
| Claude Sonnet 5.5 | low and high effort |
| Claude Opus 5 | high effort |
| Claude Sonnet 5 | high effort |
| Gemini 3.1 Pro | standard and extended thinking |
| Gemini 3.8 Flash | standard and extended thinking |
| GPT 5.6 Luna | standard and thinking |
The older Claude models ran at high effort only, so any gap to the newer ones comes from the model. Medium effort was skipped: low against high gives the clearest signal.
I have attached 6 photos of bookshelves, named a, b, c, d, e and f (in that order).
For each photo, list every book whose title you can read.
Rules:
- Only include a book if you can actually read its title in that photo. Do not guess, and do not
add books that are likely to be there but whose titles you cannot read.
- Keep the photos separate. List each book under the photo it appears in.
- If several copies of the same book are visible in a photo, list it once.
- Include the author if you can read it or are confident who wrote the book. Otherwise use null.
- Write titles as they appear on the book, in their original language and script.
Reply with only this JSON and nothing else:
{ "a": {"difficulty": "easy", "books": [{"title": "...", "author": "..."}]}, "b": ..., "f": ... }
“Attack on Titan Omnibus” is the same book
Models rarely write a title exactly as the answer key does. So titles are compared loosely: case, punctuation, accents, word order, a leading “The” and a missing subtitle are all forgiven. A pair counts as the same book above 85% similarity. Authors are scored separately and never decide a match.
A surprise: the photos got shuffled
In run 1, every Claude reply looked terrible at first, then turned out to be right under the wrong letters. Its list for “a” was really photo d. The Claude app does not show the model the filenames, so it lettered the photos in its own order. The scorer now finds the best letter-to-photo match and corrects it before scoring. Only whole lists move; a single book under the wrong photo still counts against the model.
| Model wrote | Answer key | Match? |
|---|---|---|
| Attack on Titan Omnibus | Attack on Titan: Omnibus (Volumes 1-2-3) | Yes |
| Disgusting Digestion | Horrible Science: Disgusting Digestion | Yes |
| Modern Judgements: Racine | Racine: Modern Judgements | Yes |
| You Can | You Can Win | No |
| Harry Potter | Harry Potter and the Goblet of Fire | No |
Every match and non-match is saved, so any number on this page can be checked book by book.
Four ways to get a bookshelf wrong
Real examples from the replies. Struck through is what the model wrote; after the arrow is the book on the shelf.
Right author, wrong book
The model reads the author, then names their better-known title.
The Magic of the Lost Temple→The Magic of the Lost Earrings · 5 runsBehave→Determined · SapolskyBit of a Blur→All Cheeses Great & Small · Alex James
Missed by every model
In run 1, 41 of the 295 books were found by none of the 12 setups.
- Hindi titles · जादू, मेघनाद, डेरे वाले
- Upside-down or thin spines · Silas Marner, Hunted
- Partly hidden covers · The Great Gatsby, Wild Fictions
Too vague to count
Part of the title, not enough to say which book.
Harry Potter→Harry Potter and the Goblet of FireDiary of a Wimpy Kid→Diary of a Wimpy Kid: The Long HaulBritish Art→The Thames and Hudson Encyclopaedia of British Art
Marked wrong, but really there
On the shelf, but left out of the answer key as too hard to confirm.
- The Secret Life of Bees · named in 6 run-1 replies
- Jiggy McCue · 3 in run 1, 5 in run 2
- The Bitcoin Standard · 2 in run 1, 4 in run 2
Was the model rationing its answer?
Run 1 had a pattern: on the hard photos, what models named was right, but they named too few. On photo e most setups found under half the books, and every Gemini setup stopped at about 20–27 titles.
So experiment 2 sent the two hard photos one per chat, with the same rules. If the six-photo message was holding models back, they should list more.
It was, for Claude. Every Claude setup found at least as many books alone, and Opus 5.5 at low effort caught up with Opus 5.5 at high effort. That changes the reading of run 1: most of the gap between low and high effort came from the six-photo setup, not the effort setting.
Gemini barely moved. Alone it still listed only 19–25 books on photo e; it just named fewer wrong ones.
Three setups were dropped: Gemini 3.8 Flash with extended thinking, and both GPT setups, which kept failing in their apps. The chart is at the top ↑
The same twelve chats, four days later
Cost needed timing data the first chats had not kept, so all twelve setups were run again on 4 October. This time a coding agent did the clicking: Claude Code, driving the chat apps in the Brave browser through the Claude in Chrome extension. It opened private chats, chose the model and setting, attached the photos, pasted the prompt and saved the reply exactly as given. It also read each app’s own conversation data, which confirmed the model used and recorded when it started and stopped thinking.
The rerun doubled as a first look at consistency, and the results moved more than expected.
Things only the rerun showed
- ChatGPT ran Python text recognition on the photos even in standard mode, 11 code calls in one reply.
- ChatGPT’s Think mode was served by a different model,
gpt-5-6-t-mini. - Gemini 3.8 Flash with extended thinking failed once (“something went wrong”) and was retried in a new chat.
An estimate, built from what the apps reveal
Chat apps do not report tokens, so each run was priced as if the same request had gone through the API at list prices, as a low, middle and high estimate. The chart is at the top ↑
- Input: the prompt plus the six photos, by each provider’s published image-token rule. Claude: up to about 4,800 tokens per large photo. Gemini: 1,120 per photo. GPT: one token per 32×32 patch.
- Output: the saved reply, plus thinking. Thinking is hidden, so it is estimated from time: how long the model thought, times how fast the same run wrote its reply.
- ChatGPT: its Python loop re-read the photos on every turn, and its hidden reasoning is unknown, so its range is the widest.
- Hidden system prompts are not counted. Good for comparing orders of magnitude, not cents.
Same model, very different result
The answer key was drafted by Claude Opus 5.5, the model that scored best here, at medium effort in Claude Code. Working section by section, it listed nearly every readable book. In the Claude.ai chat at high effort, the same model found about 77% of the books on the hardest photo.
The model did not change, and it even had less effort. The way it was used did. The biggest factor is probably resolution: chat apps shrink large photos, and in a shrunk 24-megapixel photo a spine title can be a few pixels tall. The harness around a model can matter more than the model or its settings. This was the biggest lesson of the project.
The planned verification harness (identify, flag the unsure ones, zoom in, re-check) was not built, because it needs API access. This is an observation, not a measured result.
| Drafting the answer key | A single chat | |
|---|---|---|
| Where | Claude Code, medium effort | Claude.ai chat, high effort |
| Detail | Small sections of the full-resolution photo | The whole photo, usually shrunk by the app |
| Passes | Over a hundred sections, one at a time | One look at six photos |
| Checking | A person reviewed every book on the list | None |
What was planned against what happened
The plan had a first benchmark, then a harness experiment, then a set of “does the packaging matter?” experiments. The benchmark is done and went further than planned. Most of the packaging experiments are still open.
| Planned | Status |
|---|---|
| Frozen answer key, 6 photos | Done |
| 2–3 models, one standard prompt | Done 7 models, 12 setups |
| Scorer: precision, recall, F1, per photo | Done |
| Failure examples and a results page | Done |
| Use APIs for control | Changed chat apps only |
| All photos at once vs one per prompt | Done hard photos |
| Repeat runs for consistency | Partly 2 runs |
| Cost per run | Done estimated |
| Direct prompting vs a verification harness | Not done needs API |
| ZIP vs separate images | Not done |
| Image compression | Not done |
| PDF, Markdown, base64 and other formats | Not done |
Read with care
- Two runs at most. Scores moved by up to 0.15 F1 between runs, so small gaps between setups may be noise.
- Chat apps, not APIs. Each app may resize photos, add hidden instructions or run tools. A result means “model X in app Y on that date”.
- “Not in the answer key” is not always wrong. Some real books were left out as too hard to confirm, so precision is slightly understated for everyone.
- Loose matching can slip. It can pair two similar titles, and a Hindi title written in Latin letters does not match the same title in Devanagari.
- Six photos. Enough to see patterns, not to rank models in general.
What could come next
- A zoom harness: cut each hard photo into overlapping full-resolution crops, ask about each, merge the lists. That would measure how much of the gap is resolution alone.
- More repeat runs, to put error bars on every score.
- Compression: original against moderately and heavily compressed photos, for accuracy and cost.
- Packaging: the same photos as a ZIP, a PDF or a document.
Run it yourself
Every photo, prompt, raw reply and score is in the repository. Scoring needs only Python 3, with nothing to install.
python benchmark/evaluate.py # experiment 1 python benchmark/evaluate_single.py # experiment 2 python benchmark/estimate_cost.py # run 2 cost python benchmark/build_site.py # this page
Repository · The full story, with more detail · Experiment notes · How scoring works · Every raw reply