Internal · not for distribution

Can an AI model watch an NFL broadcast and tell us what is happening, frame by frame?

–of scorebug answers correct (score, clock, down, possession…)
–game events detected, with – false alarms
–per hour of broadcast, at one frame every 5 s
–per frame to get all 24 answers

Heads-up: the event banners run late

  • Clef sees 1 frame every 5 s, and the scorebug itself updates a few seconds after the play, so banners trail the action by 5–10 s.
  • At 1 frame per second they would land almost on the play, for about 5× the cost (around $6 per broadcast hour instead of $1.20).
  • This is the cheap demo: the sampling rate is the only thing holding the timing back.

Watch it work

The 4th quarter as broadcast, with Clef's answers drawn on as they would arrive live: each appears when Clef finished answering (about 2 s after the frame it was asked about) and holds until the next. Press T to check the answers against our labels, F for fullscreen.

Loading…
0:00 / 0:00
Space play · F fullscreen · T check vs truth · O overlay · N next event · I this strip · ←→ ±5 s in fullscreen · Footage © NFL (Jets at Bengals, 26 Oct 2025), used with permission for internal demonstration only.

Events: what it caught

Detected only from changes in Clef's answers (e.g. Jets score 32 → 38). Click a row to jump there.

Game clockEventResultTiming

Caught

  • Every score: 4 touchdowns, 2 extra points, 2 two-point conversions
  • The comeback: tie at 2:00 → lead change at 1:54 → turnover on downs
  • Overturned two-point call at 7:52, read correctly
  • 0 false alarms

Missed

  • Review at 10:24: on screen for 1 frame, the detector needs 2
  • Bengals timeout: the timeout dashes are tiny
  • Game end: the rule waited for a kneel-down

How good was it

Accuracy per question. White tick = always giving the commonest answer; a question only works if the bar is clearly past it.

■ scorebug   ■ layout   ■ scene *   ■ editorial *  ·  faded bars are soft-label questions  ·  red number = below the always-guess line

The short version

  • Reading the scorebug: solved. Scores, clock, down, message right on all 316 frames
  • On any shot: 98% for game action, close-ups and sideline alike
  • Judgement calls: weak. Highlight, ad-break, replay are close to guessing
  • Weak spots: Bengals timeouts 85%, Jets possession (11 misses)

It knows when it is unsure

Clef's stated confidenceAnswersActually correct

90%+ sure → 99% right, and that is over half of all answers. The event detector ignores the unsure ones.

How it was done

Frames
408 stills, 1 every 5 s, the whole 4th quarter
Questions
24 fixed multiple-choice, plus where the scorebug is
Clef
Frame + zoomed scorebug crop → a probability per option, ~2 s
Truth
Scorebug labels checked against the play-by-play
Events
60%+ sure, held for 2 frames. Recorded once, replayed here

Caveats

  • We made it easy: told Clef where the scorebug is, gave it a zoomed crop
  • One quarter, one game, one network
  • One AI labeller, cross-checked with the play-by-play
  • Stills only: no motion, no memory of the last frame

The asterisks *

  • 9 judgement questions have soft labels: one reader's call, spot-checked
  • Sponsor labels missed small logos, so Clef looks worse than it is
  • Kept out of the headline 98%

Left out

  • Referee signals, "has the whistle blown"
  • Exact game clock, player names, run vs pass
  • The cheaper Clef-Flash model

What comes next