Can AI Understand Music — The Evidence on Hearing, Reasoning, and Taste

AI can already pick out instruments, tempo, melodic contour, and some tonal relationships — and it can write a paragraph of criticism with accurate terminology and a coherent shape. But those two abilities do not always rest on the same chain of evidence. The first requires the model to extract musical facts from sound. The second can run on music theory, genre history, and critical discourse absorbed from text training. The stronger the language ability, the easier it is to mistake one for the other.

As of 2026, the position best supported by evidence is this: AI perceives a subset of musical facts; it remains unreliable on whole-track structure, precise temporal relationships, and complex harmony; and it can fit the average rating of a group without that amounting to taste. The bottleneck is not only whether the model has "ears" — it is which information survives after sound enters the encoder, and how much of it ever reaches the language model to shape the answer.

Multimodal AI does not hear notes — it hears a compressed representation

Publicly documented audio language models mostly share the same input chain:

audio waveform → spectrogram or music representation → audio encoder → compression and alignment → language model → text answer

Take Qwen2-Audio. Input is resampled to 16 kHz, converted to a 128-channel Mel spectrogram, and passed through a Whisper audio encoder, after which each output frame corresponds to roughly 40 milliseconds. What reaches the language model is not a symbol that says "C major" or "four-four time" or "guitar enters here." It is a continuous vector summarizing a short slice of mixed sound. The language model uses those vectors, plus the text it has already produced, to predict the next text token.

One of the main differences between models is which kind of sound the front end was trained for. SALMONN pairs the speech-oriented Whisper with the sound-event-oriented BEATs and compresses 30 seconds of audio into 88 input tokens for the language model. MERT instead adds learning objectives built specifically for music: pitch, harmony, and acoustic structure. The first tries to make one model handle speech, environmental sound, and music at once. The second is closer to an ear trained for music.

This chain has three independent bottlenecks: the encoder decides what to keep, the connector decides what to compress away, and the language model decides what to actually use. Even when pitch or timbre information still exists inside the encoder, it can be weakened during an alignment process optimized for text generation.

There is a further limit that directly constrains what these models can be used for: many general-purpose models downmix multichannel audio to mono. Google's Gemini audio documentation states plainly that multichannel input is merged into a single channel. Such a model can tell you what sounds are present, but it cannot recover the actual left-right geometry from its input. Asking it to judge panning, stereo width, or spatial placement therefore rests on a thin evidentiary base.

This is not the ceiling of audio AI. Dual-BEATs, published in 2026, encodes the left and right channels separately and reaches 97.2% accuracy on panning localization. Spatial hearing can be recovered with a dedicated architecture. Its absence in current general-purpose products is a trade-off, not a law.

Basic recognition can be strong; structural listening still is not

MUSE tested Gemini 2.5 Pro, Qwen2.5-Omni, and Audio Flamingo 3 on 10 controlled music tasks, with a human baseline of 200 people that included 6 expert musicians. The result is not "AI is uniformly worse than humans." It is a very uneven capability curve.

Gemini 2.5 Pro reached 98.33% on instrument identification, 96.67% on melodic contour, and 96.67% on rhythm matching. On tasks involving more complex relationships, chord sequence matching fell to 66.67% and time signature identification to just 46.67%. The 6 expert musicians in the same study scored 85% and 73.3% on those last two. The expert group is very small — useful for reading the shape of the capability curve, not for precise population estimates.

The spread between models is just as large. In MUSE, Audio Flamingo 3 landed exactly at chance level on 6 of the tasks, and Qwen2.5-Omni scored 23.33% on melodic contour — below the 25% you get by guessing among four options. "Multimodal model" is not a uniform capability tier; training objectives and the audio front end usually matter more than the product name.

More traditional music information retrieval tasks expose the gap more sharply. CMI-Bench v2 evaluated 11 Audio-LLMs on standard metrics for beat, key, melody, emotion, and playing technique. Outside of music captioning, general-purpose models trail task-specific systems by a wide margin: best beat tracking was 23.69 against 88.3 for the supervised reference; melody extraction 5.06 against 72.3; guzheng playing technique 3.18 against 90.0. The best R² on continuous emotion regression was 0.00 — at most equal to a baseline that always predicts the mean.

There is an important piece of positive evidence here. MARBLE, evaluating self-supervised music representations, found that MERT-330M reaches 87.9 on beat tracking and 94.4% on the NSynth pitch task. Musical information is not machine-unrepresentable. The gap appears mostly after those representations are compressed, aligned, and wired into a language model.

Changing the input format for a single model produces differences of the same order. coremusic gave Gemini 2.5 Pro the same musical tasks as waveforms and as MIDI. With MIDI input, syncopation, transposition, and chord quality came close to perfect. With audio, accuracy dropped substantially. Downstream reasoning cannot recover pitch and relationships the front end never extracted reliably.

Writing criticism is not the same as the criticism coming from this music

MuChoMusic, published in 2024, assembled 1,187 four-option music questions spanning melody, harmony, rhythm, structure, mood, and style. Its most important finding was not the scores but the control condition: when the audio was replaced with white noise or with a random song, 3 of the 5 models tested showed no statistically significant drop.

RUListening, published afterwards, went further and found that a text-only model with no audio input at all could reach 56.4% on the same benchmark, against a chance level of 25%. Not because text models listen to music, but because they can eliminate answers that violate common sense — guessing likely instrumentation from the genre, for instance.

That also exposes a flaw in early music QA benchmarks themselves: MuChoMusic's option structure often let a model rule out two clearly irrelevant answers and then guess between the remaining two. Newer evaluations are closing that shortcut. MMAR uses stronger distractors, and every model there drops noticeably when the audio is swapped for noise; in MMAU-Pro, text-only models fall to roughly 16%–30%. The sharper conclusion is not "AI isn't listening at all" but this: whenever a question can be answered without the audio, the language model takes the cheaper text route first.

Even when the final answer is right, the explanation may be reconstructed after the fact. On MUSE's syncopation task, Gemini gave the correct verdict 37 times but counted the actual syncopated positions correctly only 4 of those times. On chord quality, it identified the decisive third in 34 of 60 questions. Chain-of-thought mostly did not improve perception, and on some tasks made it worse.

Informal community testing keeps surfacing the same pattern. In 2025, a producer had Gemini 2.0 watch a DAW and monitor the audio at the same time; after the audio was muted, the model kept describing music it claimed to be "hearing," and once switched to a player with no visual cues, its instrument and genre calls got noticeably worse. That test is documented in AudioCipher's experiment. It targeted the live screen-sharing pipeline of the time and does not directly represent the audio-upload models that followed. In another case, a Gearnews reader report, the model misread reggae as indie folk, missed brass that was actually present, and commented on an acoustic guitar that was not. These are not formal benchmarks, but they point in the same direction as the language shortcuts and auditory hallucinations found in the papers.

The bottleneck looks like wiring, not missing musical representation

Two lines of evidence support the same explanation. The first is the representation models mentioned above — MERT, MusicFM and others — where beat, pitch, and key can already be read out of the encoded representation at high quality. The second comes from 2026 work on the internal pathways of audio language models. CoAT found that the closer a model gets to the text output layer, the more its hidden states are shaped for text generation, progressively losing pitch, prosody, affect, and acoustic event information. ORCA observed connector outputs collapsing toward a single direction; after constraints were applied to fix it, multi-step audio reasoning improved markedly.

Neither study is about music specifically, so neither proves that every musical error originates in the connector. But together they explain a stable phenomenon: information being present in the encoder does not mean the language output end can call on it.

Engineering already has a more reliable detour. OpenMU does not ask the LLM to perform all the auditory analysis itself. It hands chord, tempo, key, and downbeat tasks to dedicated music tools, leaving the language model to select tools and assemble the answer. Its tool-call accuracy reaches roughly 95%–100% across several tasks — a number that measures whether the right tool was invoked, not that the underlying music algorithms are 100% correct.

That maps onto a sounder product structure: let DSP and music information retrieval systems answer "what happened," then let the language model explain "what it means." Beat and key, for example, should come from dedicated BPM and key analysis rather than from asking a general-purpose chat model to guess across a whole track.

AI can score aesthetics, but "taste" is not a single total

Music aesthetics models have produced usable results. SongEval collected 2,399 complete AI-generated songs, rated by 16 annotators with musical backgrounds on dimensions including coherence, memorability, naturalness of the vocal, structural clarity, and overall musicality. Each song was rated by 4 people, and the dedicated model correlated strongly with the average human score within its own test set.

The paper is explicit about the boundary: aesthetic quality here approximates the collective consensus of trained musicians and does not represent individual taste. In experiments on the same dataset, the automatic evaluator's average Pearson correlation across the five dimensions was about 0.912 — but that is a fit to SongEval's own annotation distribution, and it cannot be extrapolated into equal agreement with any given music expert. The paper does not report inter-annotator agreement coefficients, so this study also cannot tell us how reliable "human consensus" is at its upper bound.

The larger problem is that such a model may learn shortcuts that correlate with quality without being quality. Genre Bias or Aesthetic Perception?, from 2026, audited SongEval and found model scores correlating heavily with how close a piece sits to pop: Spearman correlations of 0.88 and 0.81 on two test sets. It systematically overrates pop and underrates high-quality examples from jazz, classical, and other genres. Predicted correlations above 0.95 across the five aesthetic dimensions also suggest the model may mostly be repeating a single latent total score.

Taste behaves more like a conditional distribution: who is listening, which traditions they know, what the work is meant to do, and when the judgment is made. In our analysis of the language map of AI music, we already saw a clear cultural mismatch between training data and actual users. Under those conditions, a "musicality: 82" with no declared reference group is likely just a mainstream training distribution dressed up as a universal standard.

So "does AI have taste" should be rewritten as three separate questions:

Detached from a reference group, "correct taste" has no single answer any paper can validate. As for whether models have anything like a subjective musical experience, existing benchmarks can only measure behavior; the text of an answer cannot prove experience exists behind it.

What credible AI music analysis should look like right now

A trustworthy system has to split its output into at least three layers: observable facts, interpretive inference, and subjective evaluation. The fact layer should come from dedicated tools wherever possible, with timestamps. The inference layer needs to state its evidence and its uncertainty. The evaluation layer must declare whose standard it represents.

Verification cannot consist of one long, professional-sounding review either. More effective methods include stripping titles and metadata; injecting silence, noise, and mismatched audio; changing exactly one chord, tempo, instrument, or section order in the same piece; repeating tests; requiring every judgment to cite a timestamp; and finally, blind evaluation by musicians. Only when a model's commentary changes as expected in response to changed audio is there evidence that the judgment is grounded in the sound.

For mixing and mastering, the bar is higher still. A general-purpose, mono, speech-optimized input pipeline should not be treated as a reliable stereo measurement tool. LUFS, peaks, spectral balance, dynamic range, phase, and inter-channel correlation should be computed by DSP first. The language model is well suited to combining those measurements with reference tracks and creative intent to propose suggestions that can then be checked.

AI works well as a collaborator in music analysis and creation, and poorly as an aesthetic judge that owes no explanation. It can flag candidate problems, organize theoretical knowledge, compare versions against a stated standard, and help someone quickly test a direction inside a music creation workflow. But deciding which tension, flaw, or deviation is worth keeping still requires a stated creative intent and a subject doing the evaluating. As we discussed in our data analysis of the creator's role, tools can expand what a person is able to decide — they do not decide for that person why the work holds together.

Key references

• Qwen2-Audio Technical Report — input and alignment architecture of a public Audio-LLM.
• MERT and MARBLE — self-supervised music representations and standard task evaluation.
• RUListening — how language priors manufacture the illusion that the model is listening.
• MUSE — controlled music perception tasks with a 200-person baseline.
• CMI-Bench v2 — Audio-LLMs against dedicated music systems on traditional metrics.
• SongEval — expert aesthetic ratings for complete generated songs.
• Genre Bias or Aesthetic Perception? — genre shortcuts in aesthetic scoring.