Can a model watch a broadcast?

01 The clip and Clef's answers

Press play. Clef's answers are drawn on the clip, and all 19 are listed beside it.

matches the human label wrong, with the correct answer next to it not scored % in amber: Clef was under 60% sure faded: right, and unchanged for 10 s

00:00 / 05:30

Editorial · Clef's call, kept off the picture

Replay. Clef sees one frame every 5 s and answers in about 1.5 s; each answer holds until the next frame. Keys: F fullscreen · O overlay · I this strip · space play

truedetected

Clef's answers ·

02 Events and detection

4 of 5 launch events caught from the answers alone. One fired 73 seconds early.

No event detector was trained. An event fires when one of Clef's answers changes and holds for two frames. A hit is within 10 seconds of the real moment.

The lateness is mostly the sampling. Looking at one frame per second instead of every five, ignition, the 100 km line and fairing jettison were each caught in the exact second.

03 How good was it

89% of all answers right, but the average hides a split.

The 19 questions fall into four groups. Two work, one does not, one is unproven.

A · LayoutWorks

97%

What is on screen

Which panels, charts, maps and insets are showing.

B · NumbersWorks

100%

Reading the figures

Clock, altitude, speed, distance: 260 of 261 correct.

C · VehicleUnreliable

72%

What the rocket is doing

Boosters, engine flame and flight phase are near guesswork.

D · EditorialUnproven

87%

Highlight? Safe for an ad?

Only 6 points better than always giving the commonest answer.

Animated rocket in flight, side boosters attached, with a telemetry panel reading altitude 35 km, distance 28 km, speed 1.33 km/s

READ CORRECTLY

BOOSTERS "SEPARATED", 89% SURE.
THEY ARE STILL ATTACHED.

One frame at 2:45 shows both sides of the result.

260/261

It reads the screen.Small figures in a corner, in French number format, right on every frame but one.

73 s

It misjudges what is physically happening.It said the boosters were gone 73 seconds early, on 16 frames in a row, and was confident each time.

99%

Its confidence is mostly usable.Answers given at 90% confidence or more are 99% right, and that is 59% of all answers. Below 60% it is right less than half the time.

$0.50

It is cheap.Per hour of video at one frame every 5 seconds, about 1.5 seconds per frame.

Bottom line. Ready to trial for reading on-screen graphics such as scores, stats and panels. Not ready to judge the action itself.

04 How it was done

Sample, ask, check, detect.

Clef is not a chatbot. It takes a picture and a list of multiple-choice questions and returns a probability for every option in one pass, so there is no prose to parse and every answer carries a confidence.

Sample66 frames from a 330-second clip, one every 5 seconds.

AskThe same 19 questions per frame. Each has an escape option such as "can't tell".

CheckScore against labels drafted by a model, then reviewed and corrected by a person.

DetectA new answer at 60% confidence or more, held for two frames, becomes an event.

The 19 questions · hover for the wording

Every answer on one page

19 questions down, 66 frames across. Brighter means more confident, so bright red is confidently wrong. The right-hand column compares accuracy with always giving the commonest answer. Hover for the frame, click to jump the clip there.

right, confident right, unsure wrong, unsure wrong, confident not scored

Does its confidence mean anything?

Group every answer by how confident Clef was, then check how often each group was right. The top group is right 99% of the time. That gives a workable rule: act on the confident answers automatically and send the rest to a person. The booster errors are the exception: they sit at 80–95% confidence.

05 Caveats

Read the numbers as an indication, not a benchmark.

One clip, one labeller.

One broadcast, 66 frames, labels checked by one person. A single frame moves any question by 1.5 points.

The booster error would not move.

Rewording the question, sampling every second and giving the model its previous answers all left it in place. It is a model error, not a setup error.

Editorial judgement is unproven.

"Is this a highlight?" and "is it safe to cut away?" beat the commonest answer by about 6 points. Not enough to automate an editor's call.

Liftoff is called late.

Clef keeps saying "on the pad" until the rocket is well clear, so liftoff is 9 to 10 seconds late even at one frame per second.

Speed was measured once.

About 1.5 seconds per frame, on one afternoon, through one provider.

The footage is not ours.

The broadcast includes partner graphics. Anything client-facing should use footage the company owns.

Wording matters more than expected

Close-up of the telescope's foil with a small inset animation of the rocket at bottom right

CLEF ANSWERED ABOUT THIS INSET

Late in the clip the rocket is not in the main picture, but Clef kept answering "in flight, boosters separated". It was describing the small animation in the corner. We had never told it to ignore the insets, so that failure was our question, not the model.

Rewording fixed it and broke a different question, which took one more rewording to recover. Every new use needs its questions tuned against labelled frames first.

83%"Telescope released?" original

50%after first rewording

88%final wording

06 What we tried and dropped

Cheaper, smarter, denser. Only denser paid off.

Layout questions only

The small model, Clef-Flash.

A sixth of the price, eight points worse, and no faster.

Overall accuracy	88.6% → 80.9%
Editorial questions	87% → 51%
Cost for the clip	$0.046 → $0.007

Dropped

Telling it its previous answer.

No gain, and slightly worse exactly where the picture had just changed.

Overall accuracy	88.6% → 87.5%
When the truth just changed	76% → 70%
False alarms	1 → 2

Keep for key moments

One frame per second.

Timing to the second, at five times the cost. Boosters still missed.

Ignition	+9 s → 0 s
Fairing jettison	+2 s → 0 s
Cost per minute	$0.008 → $0.042

Replaced

Spotting ignition from the clock.

The broadcast hides the countdown from 0:50 to 2:10, so this fired 69 seconds late. "A flame appears" works.

Avoided

Forcing a guess.

Every question has a "can't tell" or "not visible" option, so the model can decline instead of inventing an answer.

Not run

A second kind of footage.

Planned, but it needs a clip the company owns. This is the next step.

Event timing under each setup

07 What comes next

Run it on footage we own, on the questions it is good at.

Next stepThe same pipeline on a clip the company owns, in the domain of most commercial interest, with questions limited to what is printed or laid out on screen.

Live sports overlaysLooks feasible

Reading a scoreboard or on-screen graphic is the proven strength. Detecting the action itself, a wicket or a goal, is the booster question again and should be assumed weak until tested.

Video archivesLooks feasible

Knowing what kind of shot is on screen supports automatic chaptering and "find every clip showing…" search.

Content complianceWorth a trial

Auto-accept the 59% of answers given at high confidence and review the rest by hand.

Highlights and ad breaksNot ready

The editorial questions barely beat the commonest answer.

Cost is not the obstacle: all six runs behind this page, 519 model calls, came to 32 cents.