Can AI Hear AI Music?

01 The experiment

We played each chatbot real songs and asked: AI or human?

STEP 1

Pair the songs

Each AI song paired with a human song in the same style, so genre can't give it away.

STEP 2

Upload, one per chat

To ChatGPT, Claude and Gemini. A fresh chat every time, no memory.

STEP 3

Describe, then decide

First "describe this file", then "AI or human, and how sure are you?"

STEP 4

Score

131 verdicts checked against the truth.

AI-made

15songs

Suno and Udio, from public datasets

Suno v5 / v5.5 (current)Suno v3.5 (older)Udio

vsmatched
by style

Human-made

10songs

Creative Commons, released before AI music existed

little-known artistssame genresno famous songs

Five experiments, from easy to hard

Each experiment changes one thing, so we can see what a model is relying on.

EXP 1

The original file

Left to go on:everything, incl. tags

EXP 2

Cleaned file, full song

Left to go on:sound, length, loudness

EXP 3

30-second clip

Left to go on:only the sound

EXP 4

Easier AI: older generators, with vocals

Left to go on:sound + AI-written lyrics

EXP 5

As 4, lyrics written by people

Left to go on:only the sound

1 → 3: the same 5 AI + 5 human songs, giveaways removed one by one4 → 5: new songs, chosen to make AI easier to spot

02 Results

Correct verdicts out of 10 songs

Click any cell to see every song, the model's verdict and its reason.

Experiment

ChatGPT

Claude

Gemini

"Detection" is reading the label. ChatGPT and Claude open the file with code and find made with suno. Remove the tags and they drop straight to chance, answering "AI" for almost everything.
Gemini is the only one that listens, and it scores about 6/10 at every level, leaning "human" with stock phrases like "clean transients, natural groove".
Experiment 4 looked like a breakthrough: Gemini ranked every AI song above every human one. But it cited the lyrics each time, and they were AI-written. With human lyrics (experiment 5) it fell to chance.
ChatGPT sounds as if it listens but doesn't: it said "no vocals" on 7 of 10 vocal songs.
High frequencies didn't help: a human song at full quality was still called AI by all three (side test).

05 How it was done

Method in detail

What each experiment did to the files

Exp	Songs	File processing	Models
1	5 Suno v5/v5.5 + 5 human	Original file, only renamed to a random ID. Tags still say `made with suno` or name the artist and year.	All 3
2	same 10	Tags stripped; every file re-encoded identically (44.1 kHz, 192 kbps MP3).	All 3
3	same 10	30 s from a quarter of the way in, loudness-matched, short fades, 15 kHz low-pass.	All 3
4	5 Suno v3.5/Udio with vocals + 5 human with vocals	30 s clips at 16 kHz mono (the only quality the AI dataset offers); human songs given the same compression so quality matches.	All 3
5	5 Suno v3.5 with human-written lyrics	Same as 4. Scored against Gemini's answers on the same 5 human clips from experiment 4.	Gemini

A pilot on the same 10 songs had already tried the experiment-3 condition in single-turn form, including full songs and ChatGPT's thinking mode. Everything landed at chance, which led to this graded design.

The two-turn chat

Turn 1 · describe

AI is not mentioned. Tempo, instruments, vocals, length, sample rate, tags, and how the model accessed the audio.

Gemini, AI song, exp 1:
  accessed: "listened directly"
  length: 156 s   (actual 248 s)
  tags: null

Turn 2 · decide

Same chat: "AI" or "human", confidence 50–100%, main basis, up to five reasons.

Same song:
  HUMAN, 88%, "musical content"
  "authentic groove, natural humanization"
ChatGPT on it: AI, 99%, "made with suno" tag

Settings: ChatGPT temporary chat, Think off · Claude Sonnet 5.5, medium effort, memory off · Gemini Flash, temporary chat.

Can the models perceive the file? (turn 1, experiments 1–3)

Model	How it got at the audio	Sample rate right	Length error	"Could not access"
ChatGPT	Python code (ffprobe, librosa)	30 / 30	0 s	0
Claude	Python code; says it cannot listen	30 / 30	0 s	0
Gemini	Listens; no code	1 / 30	90–130 s	6 / 30

Gemini's "could not access" answers clustered on Suno files, repeated on retry, and are scored as given (50%).

Side test: do high frequencies sound "human"?

One AI song (nothing above 8 kHz) and one human song (full range), otherwise identical.

Model	AI song	Human song	Used the difference?
Claude	AI 57%	AI 55%	Measured it, moved 2 points
ChatGPT	AI 58%	AI 58%	Didn't measure it
Gemini	AI 92%	AI 88%	No mention

Time and cost per chat

Model	Time	Tokens (estimated)	Illustrative cost
ChatGPT	≈ 2 min	≈ 2,500 in · 2,500 out	≈ $0.045
Claude	≈ 1–1.5 min	≥ 1,200 in · 650 out	≥ $0.013
Gemini	≈ 40 s	≈ 2,900–19,000 in · 350 out	≈ $0.013–0.06

The apps hide token counts, so tokens are estimated from page text (~4 characters per token). Gemini tokenises the audio itself (32 tokens per second), so full songs cost more than clips. Cost uses one illustrative rate ($3 / $15 per million tokens in / out), not any provider's prices.

Caveats

Small10 songs per experiment, one run each. A smoke test, not a precise measurement.

Consumer appsThe app picks the model version, and it can change without notice.

Our processing shows upIn experiment 3, Claude cited the low-pass and fades our clean-up adds to every clip. They are identical for AI and human, so they can't help, but they distort the reasoning.

Experiments 4–5All audio is 16 kHz mono; experiment 5 is all Suno (the dataset has no Udio songs with human lyrics).

Reproduce

Build the files: scripts/make_levels.sh (exp 1–3), make_vocal_round.sh (4), make_vocal_hf.sh (5), make_freq_test.sh (side test). Prompts in prompts/.
Run the chats: scripts/browser/. Score: score_ladder.py, score_vocal.py. This page's data: build_page_data.py → docs/data.js.
Every answer, both turns: results/*_raw_log.jsonl. Full audio and answer keys are not in the repository; only the four excerpts above are.

Can ChatGPT, Claude or Gemini tell whether a song was made by AI?