1 the problem

What can an audio embedding model actually hear, and can typing a word find it?

What the model knows vs what a word finds

2 the models

Two AI models that turn sounds and words into numbers

Both turn a clip and a sentence into numbers (an embedding). Similar things land close, so words can search sound.

EmbeddingGemma 2 (Gemma): a words model with ears

  • Google's text model with an audio "ear": audio becomes pseudo-words.
  • Strong on speech and meaning; the podcast search runs on it.

CLAP: a sound model that learned captions

  • LAION's model trained on sounds paired with captions ("a dog barking").
  • Built to match sounds to descriptions; hears the voice itself.

Why two? A second opinion: shared results are general, the rest are quirks. They turned out good at different things.

What the numbers look like: a map of every clip

Every clip, by sound family (Gemma)

How to read: each dot is one clip, flattened from the model's numbers onto a 2-D map (UMAP). Colour = sound family. No labels were used to place the dots: similar sounds cluster on their own. The scattered clumps are emotional speech; each mixes all six emotions, so they group by something else (likely the sentence spoken).

Clips and typed words on one map (Gemma)

How to read: yellow rings = typed words, dots = clips. Words clump together instead of mixing with the clips they describe (the "modality gap"). Hover to see labels.

3 the datasets

Public datasets where every clip already has a correct label

4 results

What we measured, and the charts

Probe = "is the information in the model?"

A tiny classifier trained on the model's numbers, tested on unseen clips from new people. The model itself is unchanged.

Zero-shot = "can a typed word find it?"

No training: each clip gets the label of the nearest typed word, like a search box.

Probe high, zero-shot low = "knows but can't name". Always compare against guessing.

The overview: is it in the model vs can a word find it

Every sound class, one dot each

Gemma

CLAP

How to read: each dot is one sound (dog, rain, giggle…). Right = the probe finds it (it's in the model). Up = typing its name finds it. On the dotted line, both agree; far below the line = the model knows the sound but its name doesn't find it. Hover a dot to see which sound.

Fine distinctions: sets of similar sounds

How to read: each row is a set of similar sounds the probe must tell apart (e.g. giggle vs chuckle). The dot is the probe's accuracy (blue = Gemma, orange = CLAP). The horizontal line through a dot is the range we're 95% sure the true accuracy lies in (shorter = more certain). The grey tick is guessing for that set. The further a dot sits to the right of its grey tick, the finer the hearing.

Emotion: who, what or how

Bars = how far above guessing a probe gets (0% = guessing, 100% = perfect) at telling the emotion, the sentence or the speaker. "Prosody" = simple loudness/pitch/speed measurements, for comparison.

Which emotion the Gemma probe picks (rows = the true emotion). The diagonal is correct.

Music: can a genre word find the genre?

Type the genre: the share of the nearest tracks that really are that genre (bar), against what random picking would give (yellow tick).

Per genre: filled dot = in the model (probe), ring = typed word finds it.

What makes classical "classical"? We edited the audio and re-measured

pick another genre

Change one thing at a time

How to read: each row = one edit. Further left = more of the genre signal lost; −100% = all of it.

Which sound qualities go with the genre score

How to read: bar right = more of this quality, higher genre score. Bar left = more of it, lower score.

Same instrument, different genre

Words nearest the genre

Mistaken for the genre

Back to the podcast search

5 findings

What it all means

Finding 1: Gemma hears sounds well, but words barely reach them. CLAP does both.

    Finding 2: laughter types are only roughly separable, and not "better" than birds

      Finding 3: emotion is in there, but unnameable

        Finding 4: genre words partly work; "classical" means tonal sound without drums

            Finding 5: the podcast search's failures sit where the model is weak

                6 conclusions

                Which model is better? It depends on the job

                Is anything deployable? Yes, a few concrete things, all cheap

                That is the result. Everything below is the how.
                Method glossary, detail and limits

                Glossary

                Dataset details and licences

                Technical summary

                Resolution by family

                Click a row to see its classes.

                Effective vocabulary per family

                    Words vs sounds, in detail

                    Synonym clouds

                    Each synonym's nearest sound clips: share that are the right sound.

                    Centring strength vs zero-shot accuracy (Gemma)

                    Same shape? Audio-centroid vs text similarity

                    The gap per family

                      Emotion and music, in detail

                        Regions (instrumental)

                        Instrumental filter

                            How it was measured

                            Protocol

                              Limits

                                Sanity checks

                                Reproduce