Both turn a clip and a sentence into numbers (an embedding). Similar things land close, so words can search sound.
Why two? A second opinion: shared results are general, the rest are quirks. They turned out good at different things.
How to read: each dot is one clip, flattened from the model's numbers onto a 2-D map (UMAP). Colour = sound family. No labels were used to place the dots: similar sounds cluster on their own. The scattered clumps are emotional speech; each mixes all six emotions, so they group by something else (likely the sentence spoken).
How to read: yellow rings = typed words, dots = clips. Words clump together instead of mixing with the clips they describe (the "modality gap"). Hover to see labels.
A tiny classifier trained on the model's numbers, tested on unseen clips from new people. The model itself is unchanged.
No training: each clip gets the label of the nearest typed word, like a search box.
Probe high, zero-shot low = "knows but can't name". Always compare against guessing.
How to read: each dot is one sound (dog, rain, giggle…). Right = the probe finds it (it's in the model). Up = typing its name finds it. On the dotted line, both agree; far below the line = the model knows the sound but its name doesn't find it. Hover a dot to see which sound.
How to read: each row is a set of similar sounds the probe must tell apart (e.g. giggle vs chuckle). The dot is the probe's accuracy (blue = Gemma, orange = CLAP). The horizontal line through a dot is the range we're 95% sure the true accuracy lies in (shorter = more certain). The grey tick is guessing for that set. The further a dot sits to the right of its grey tick, the finer the hearing.
Bars = how far above guessing a probe gets (0% = guessing, 100% = perfect) at telling the emotion, the sentence or the speaker. "Prosody" = simple loudness/pitch/speed measurements, for comparison.
Which emotion the Gemma probe picks (rows = the true emotion). The diagonal is correct.
Type the genre: the share of the nearest tracks that really are that genre (bar), against what random picking would give (yellow tick).
Per genre: filled dot = in the model (probe), ring = typed word finds it.
How to read: each row = one edit. Further left = more of the genre signal lost; −100% = all of it.
How to read: bar right = more of this quality, higher genre score. Bar left = more of it, lower score.
Each synonym's nearest sound clips: share that are the right sound.