I started making pop and applied music for the mainland Chinese market in 2017, mostly as a composer, arranger and producer, for clients in film, TV and games as well as ordinary pop listeners. In 2021 I joined Professor Zhiyao Duan’s lab at the University of Rochester, where I started with busy work and slowly moved toward deep learning; I now lead research and labs at Smule, working with a lean team to figure out which models to train and how to train them. Since 2025 I’ve put less and less of my energy into music production, and these days I mostly write demos for friends and rarely do the arranging myself or take a song through the full production process.

I have a lot going on now, so the time I can spend focused on one song is probably under an hour. The first half hour goes to the melody, lyrics, chords and a rough groove. I’m used to writing at the keyboard, and I sketch these out quickly with a piano or guitar sound. I’m much faster than a model at this step, and making the demo also makes it clearer to me what I want. After that, I take ten minutes or so to lay out the other instruments I want and roughly where they should go. I usually use Logic Pro X, because I know it well and its stock instruments are reasonably good.

Then I turn to a music generation model like Suno and, mostly by editing the text description, have it generate the overall result I’m after. Here the model mainly helps me swap instruments, try different instrumentations, and make a demo I didn’t sing very carefully sound better; where I haven’t fully worked out what I want myself, I use it to try a few more versions until I find it. I usually pick a few versions, download all the stems, edit and remix them in Logic, and hand that in.

Unfortunately, what these models produce still can’t be handed in as a finished arrangement and production; compared with where language models and video models are today, the gap is fairly large. In this post I want to make a guess: the gap exists mainly because music generation models need finer-grained text descriptions. To write such descriptions, a model first has to understand the music it hears, and the data needed to train that understanding is something we don’t have yet.

Models keep getting bigger and the metrics keep going up, but they don’t sound better

I often hear two views in the industry: one says the capabilities of music generation models scale easily, the other says they don’t.

If the goal is only to fit the pretraining data better, or to fit some set of human preferences better, there’s no reason it shouldn’t scale, and by and large it does. YuE compared three sizes, 0.5B, 2B and 7B, and in human evaluation, the larger the model and the training budget, the more its musicality was preferred⁠[1] YuE: Scaling Open Foundation Models for Long-Form Music Generation. Yuan et al., 2026.. MusicGen reported three sizes, 300M, 1.5B and 3.3B: perplexity fell from 56.1 to 48.4 to 46.1 and FAD from 0.96 to 0.86 to 0.82, both improving steadily⁠[2] Simple and Controllable Music Generation. Copet et al., 2023.. Mikey Shulman, the CEO of Suno, also said in a 2024 interview that their models scale much like text models do, and that “bigger is usually better”⁠[3] Making Transformers Sing - with Mikey Shulman of Suno. Latent Space, 2024..

But for the same MusicGen models, the overall quality rated by humans (OVL) is 78.3, 81.9 and 79.2, and it stops rising at 1.5B. The authors say so themselves: subjective quality no longer improves past 1.5B, and the 3.3B model only seems to follow the text prompt more closely⁠[2] Simple and Controllable Music Generation. Copet et al., 2023.. There is a similar observation in speech: Cuervo and Marxer found that the linguistic ability of speech-only language models scales up to three orders of magnitude more slowly than that of text models⁠[4] Scaling Properties of Speech Language Models. Cuervo & Marxer, 2024.. Audio simply asks more of the data and the supervision signal.

I think both views are right. A bigger model does fit better, but judging from the human ratings above, it doesn’t necessarily sound better. Below are the three problems I run into most often in day-to-day use. I think they affect the usability of these models to a large degree, and that the solution behind all three is the same: finer-grained text descriptions.

The model only gets the gist of my text instructions

As I said at the start, I mostly get the result I want out of the model by editing the text description. The model should be able to understand the instruction I give it and use it to narrow the search space of its outputs; a good instruction is one that pins the model down. In image and video, we can require the output to satisfy certain pixel-level constraints, such as a luminance map, a depth map, normals derived from an untextured 3D model, or the identity of a particular character. If I paste in a depth map, the model follows that depth map; if I ask for a fixed character, the model keeps that character.

In music generation, though, we often can’t get control that direct. Some controls are direct, like melody, lyrics and singer identity; some are more or less direct, like instruments and groove; and some are very vague and coarse, like style or a certain emotional feel. To some extent, a creator’s skill lies in turning vague controls into direct ones, step by step.

Research and products are both making control more concrete, bit by bit. Instruct-MusicGen lets the model add, remove or extract a single stem when instructed⁠[5] Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning. Zhang et al., 2025.; Suno Studio can generate stems and export them as audio or MIDI⁠[6] Introducing Suno Studio. Suno, 2025.. But as soon as the instruction gets vague, the results fall off quickly. RIME has multimodal LLM agents do mixing and post-production from instructions: when the instruction states the effects and parameters directly, the edit similarity is 0.718 for GPT-Audio-Mini and 0.687 for Gemma 3n; when it only describes the overall feel, the two drop to 0.519 and 0.528 respectively⁠[7] RIME: Enabling Large-Scale Agentic Music Post-Production. Schaffer & Singh, 2026..

Without this kind of fine-grained, direct control, I can’t really control what a music generation model outputs.

Instruments change while they’re playing

The second problem is that I often hear instruments change on their own in the generated result. The most typical case is a brass section blending into a string section that plays a similar role, with instruments that should stay on one melodic line jumping around; the vocal comes and goes, or suddenly turns into another instrument. The drum sound and the soundstage of the song often jump as well. Subtler jumps happen in almost every generation: I hear the size of the guitar, the recording conditions and the texture of the strings changing constantly. If I ask for a single brass instrument, what I get is often a continuous “spectrum of brass instruments” that slides from trumpet all the way to tuba, with some chance of sliding on to cello, and the cello may well slide on to double bass.

Papers report similar things. The SongGen authors note that when the mix is used directly as the training target, the model tends to prioritize the more predictable accompaniment and often neglects the vocal features⁠[8] SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation. Liu et al., 2025.. A challenge paper trained on only 457 hours of data found that distinctive instruments such as saxophone, harmonica and vibraphone are often rendered as generic timbres⁠[9] Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches. Junyoung Koh, 2026..

I suspect this has to do with how the model represents audio. A very common practice in the industry now is to first compress audio into more “semantic” discrete tokens, which are easier for a language model to model. The approach comes from speech research: MusicLM largely keeps the structure of AudioLM, using w2v-BERT to produce semantic tokens and SoundStream to produce acoustic tokens⁠[10] MusicLM: Generating Music From Text. Agostinelli et al., 2023..

In music, the line between semantic and acoustic is not that clear, but people still train this way, and each group makes its own assumption about what counts as semantic. MERT uses an RVQ-VAE as the acoustic teacher and adds a CQT-based musical teacher⁠[11] MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training. Li et al., 2024.; MuQ predicts tokens produced by Mel-RVQ⁠[12] MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization. Zhu et al., 2025.; MuCodec first does masked language modeling, then adds mel-spectrogram and CQT reconstruction along with a lyric transcription loss made up of CTC and RNN-T⁠[13] MuCodec: Ultra Low-Bitrate Music Codec. Xu et al., 2024.. The MuSeC authors put it more bluntly: these semantics-guided schemes were designed for speech and don’t fit music well⁠[14] Rethinking Music Tokenization: A Semantic Codec toward High-Fidelity LLM Music Generation. Chen et al., 2026..

But what exactly is “semantics” in music? Nobody seems to have ever defined it clearly. If we train with masked prediction, we are in effect saying that the target that gets masked out for the model to predict is the semantics; unfortunately, we often don’t know whether that target contains all the semantic information it should. If we instead ask the tokenizer to learn to output text descriptions and lyrics from audio, then the boundary of the semantics is drawn by those descriptions and lyrics.

My guess is that this is related to the jumps I hear. If the line itself is not drawn clearly, things that are “semantically” close, such as trumpet and tuba or cello and double bass, may be treated by the model as roughly the same thing and lumped together.

I think one reason this approach is enough for speech but not for music is that information is structured differently in the two. Speech mostly has a single speaker whose identity is stable, and the information basically changes along a single axis, time. Music has both a horizontal and a vertical dimension: vertically, several parts are stacked at the same moment, with sources that overlap heavily and are much harder to pull apart than in speech⁠[15] Discrete Audio Tokens: More Than a Survey! Mousavi et al., 2025.; horizontally, the function of each part keeps changing over time.

Everything it generates sounds about the same

The third problem is mode collapse, which I can observe very easily. However I change the prompt, what comes out is about the same; the model is too opinionated. Put simply, RL has been overdone. The odd thing is that it also looks as if RL hasn’t been done enough: control is still poor, yet the output is already very homogeneous, somewhat like the collapse we saw in the GAN era.

A paper from August 2026 happened to measure exactly this⁠[16] The Algorithmic Flattening of Sound: Computational Evidence and Justice Implications of AI Music Homogenization. Slendebroek & Metaxa, 2026.. The authors analyzed the output of Suno and Lyria 3 across four genres with 72 MIR features and found that Lyria 3 has lower diversity within a genre, while Suno flattens the differences between genres. Neither system follows the prompt faithfully, and an ordinary classifier can separate AI tracks from human tracks almost perfectly.

There is already a fair amount of evidence that RLHF reduces diversity in music. SMART tuned a symbolic music generation system using Audiobox Aesthetics as the reward and found that over-optimization sharply reduces output diversity; after optimization, the outputs often score higher on this reward than real performances do, which, as the authors themselves say, only shows the limitation of using this reward as a proxy metric⁠[17] SMART: Tuning a symbolic music generation system with an audio domain aesthetic reward. Jonason et al., 2025.. DRAGON found that optimizing per-song FAD worsens Vendi diversity by 17.7% on average⁠[18] DRAGON: Distributional Rewards Optimize Diffusion Generative Models. Bai et al., 2025.. On language models, Kirk et al. likewise found that RLHF output is markedly less diverse than SFT output⁠[19] Understanding the Effects of RLHF on LLM Generalisation and Diversity. Kirk et al., 2024..

Beyond RL, the classifier-free guidance commonly used at inference also pushes diversity down: on a MusicLM-based model, raising the guidance scale from 1 to 7 drops the diversity score from 0.5 to 0.3⁠[20] Diversity-Rewarded CFG Distillation. Cideron et al., 2025..

Part of RLHF’s diversity problem comes from the mode-seeking behavior of RL itself. GX-Chen et al. go a step further and point out that under common settings (weak regularization, or all correct answers receiving the same reward), the optimal target distribution of KL-regularized RL is often unimodal to begin with, so the collapse of diversity is in effect written into the objective⁠[21] KL-Regularized Reinforcement Learning for Generative Modelling is Designed to Mode Collapse. GX-Chen et al., 2026..

Derivation: why is the optimal distribution unimodal?

Setup: Let π\pi be the model being trained, πref\pi_{\text{ref}} the reference model (usually the model before RL starts), R(y)R(y) the reward of an output yy, and β>0\beta > 0 the regularization strength. KL-regularized RL (here meaning the most common form, reverse-KL regularization) maximizes

Jβ(π)=Ey∼π[R(y)]−β DKL(π ∥ πref).(1)J_\beta(\pi) = \mathbb{E}_{y \sim \pi}\big[R(y)\big] - \beta\, D_{\mathrm{KL}}\big(\pi \,\|\, \pi_{\text{ref}}\big). \tag{1}

Optimal solution: Expanding the KL term by its definition and absorbing the reward into the logarithm,

Jβ(π)=Ey∼π[R(y)−βlog⁡π(y)πref(y)]=−β Ey∼π[log⁡π(y)πref(y) eR(y)/β]=−β DKL(π ∥ Gβ)+βlog⁡Zβ,\begin{aligned} J_\beta(\pi) &= \mathbb{E}_{y \sim \pi}\Big[R(y) - \beta \log \frac{\pi(y)}{\pi_{\text{ref}}(y)}\Big] \\ &= -\beta\, \mathbb{E}_{y \sim \pi}\Big[\log \frac{\pi(y)}{\pi_{\text{ref}}(y)\, e^{R(y)/\beta}}\Big] \\ &= -\beta\, D_{\mathrm{KL}}\big(\pi \,\|\, G_\beta\big) + \beta \log Z_\beta , \end{aligned}

where

Gβ(y)=πref(y) eR(y)/βZβ,Zβ=∑y′πref(y′) eR(y′)/β.(2)\begin{aligned} G_\beta(y) &= \frac{\pi_{\text{ref}}(y)\, e^{R(y)/\beta}}{Z_\beta}, \\ Z_\beta &= \sum_{y'} \pi_{\text{ref}}(y')\, e^{R(y')/\beta}. \end{aligned} \tag{2}

ZβZ_\beta does not depend on π\pi, and the KL divergence is non-negative and zero only when the two distributions are identical, so (1) has exactly one optimal solution: π∗=Gβ\pi^* = G_\beta. In other words, even with unlimited compute and perfect optimization, RL will only ever arrive at GβG_\beta. It is the “target distribution” of this objective.

Probability ratio: Take any two outputs y1y_1 and y2y_2. The normalizing constant ZβZ_\beta cancels in the ratio, so

log⁡Gβ(y1)Gβ(y2)=log⁡πref(y1)πref(y2)+R(y1)−R(y2)β.(3)\log \frac{G_\beta(y_1)}{G_\beta(y_2)} = \log \frac{\pi_{\text{ref}}(y_1)}{\pi_{\text{ref}}(y_2)} + \frac{R(y_1) - R(y_2)}{\beta}. \tag{3}

The first term on the right is the reference model’s existing preference, and the second is the reward difference divided by β\beta. Which of the two outputs gets more probability in the target distribution is determined entirely by these two terms.

Case one: all correct answers have the same reward. This is the standard setup for verifiable tasks, with 1 for a correct answer and 0 for a wrong one. For any two correct answers, the second term of (3) is zero, so

Gβ(y1)Gβ(y2)=πref(y1)πref(y2),\frac{G_\beta(y_1)}{G_\beta(y_2)} = \frac{\pi_{\text{ref}}(y_1)}{\pi_{\text{ref}}(y_2)},

and this ratio does not depend on β\beta. RL only moves probability from wrong answers to correct ones, while the relative proportions among the correct answers are carried over from the reference model unchanged: whichever kind of answer the reference model favored, the target distribution favors to the same degree, and lowering β\beta doesn’t help. As long as the reference model already assigns these answers very different probabilities, the target distribution is concentrated on one of them in the same way. Replacing the regularizer with forward KL doesn’t change this conclusion.

Case two: the rewards differ and the regularization is weak. Consider first two outputs with the same reference probability, for which (3) becomes

Gβ(y1)Gβ(y2)=exp⁡ ⁣(R(y1)−R(y2)β).\frac{G_\beta(y_1)}{G_\beta(y_2)} = \exp\!\Big(\frac{R(y_1) - R(y_2)}{\beta}\Big).

A linear gap in reward becomes an exponential gap in probability. The paper’s example: with rewards that differ by only 0.1 and β\beta set to the commonly used 10−310^{-3}, the ratio is e100e^{100}, a factor of more than 104310^{43}. The conclusion is the same when the reference probabilities differ: the first term of (3) is fixed while the second keeps growing as β\beta shrinks, so for small enough β\beta, the output with the slightly higher reward always ends up with the overwhelming majority.

Condition for multimodality: For two outputs to have the same probability in the target distribution, the right-hand side of (3) must be zero, that is,

R(y2)−R(y1)=βlog⁡πref(y1)πref(y2).(4)R(y_2) - R(y_1) = \beta \log \frac{\pi_{\text{ref}}(y_1)}{\pi_{\text{ref}}(y_2)}. \tag{4}

The reward difference has to exactly cancel the difference in log probability under the reference model. β\beta is only a knob here: turned up, it favors the outputs the reference model likes; turned down, it favors the outputs with higher reward. As long as the two outputs have different rewards, there is at most one β\beta that makes them exactly even. Common settings generally don’t satisfy this condition, so most of the probability in the target distribution sits on one mode rather than being shared among all the high-quality outputs. The paper calls the latter case multimodal, and the former is what I call unimodal here. This collapse isn’t caused by insufficient exploration or poor optimization: the better the optimization, the closer the model gets to GβG_\beta, and the more complete the collapse.

The other part comes from the reward model: we have no way to inspect what it has actually learned or what it is rewarding. So the most common practice is to watch the RL process, pull out some checkpoints and listen to them ourselves. Listening this way, it’s hard to hear a lack of diversity, and we can only hear some problems with how the rewards are set up; once a problem is found, the usual practice is to add another reward, and then another.

Rewards patched together like this can easily affect generalization. The DiffRhythm 2 authors write that as the number of preferences grows, model merging often noticeably lowers the average performance across them⁠[22] DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching. Jiang et al., 2025.. TuneJury found that while the reward goes up, MAD, which measures distributional fidelity, gets worse: with expert iteration at a learning rate of 1e-5, the reward rose by 0.416 and MAD worsened by 0.669, which the authors call a classic signature of reward exploitation⁠[23] TuneJury: An Open Metric for Improving Music Generation Preference Alignment. Kim et al., 2026..

More fundamentally, everyone’s taste in music is different. MusicRL collected 300,000 pairwise user preferences and found that text adherence and audio quality account for only part of them, and that fine-tuning with the quality reward alone is also prone to over-optimization⁠[24] MusicRL: Aligning Music Generation to Human Preferences. Cideron et al., 2024.. CMI-RewardBench re-annotated 500 test samples: Krippendorff’s α between annotators was only around 0.50, and Gemini-2.5-Pro’s agreement with human preferences also struggled to exceed 80%⁠[25] CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction. Ma et al., 2026..

The aesthetic data and metrics we have all record some kind of consensus taste. SongEval had 16 professional annotators with music backgrounds rate 2,399 full songs on 5 dimensions⁠[26] SongEval: A Benchmark Dataset for Song Aesthetics Evaluation. Yao et al., 2025.; Audiobox Aesthetics breaks aesthetics down into four dimensions: production quality, production complexity, content enjoyment and content usefulness⁠[27] Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound. Tjandra et al., 2025.. Xiao et al. also show theoretically that KL-regularized RLHF can, in extreme cases, exhibit preference collapse, in which minority preferences are almost entirely disregarded⁠[28] On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization. Xiao et al., 2025..

Even in text, where data is plentiful, investment is heavy and research resources are ample, this problem hasn’t been fully solved. People have simply broken the reward into finer and more objective pieces, and put most of their training effort elsewhere.

How does vision do it?

As I said at the start, music generation models are well behind video models today, so I want to look first at how the vision side does it. For some time now, the vast majority of vision research teams have put most of their effort into one thing: finer-grained text descriptions.

At training time, they use a model to rewrite the descriptions of the training data, and this ability to write descriptions can borrow directly from the prior of text models. DALL·E 3 trained a dedicated captioner to write descriptive synthetic captions and ended up training on 95% synthetic captions plus 5% original ones; the paper also notes that GPT-4 can “upsample” any short caption into a highly detailed description⁠[29] Improving Image Generation with Better Captions. Betker et al., 2023.. PixArt-α relabeled LAION images with LLaVA, which raised the number of nouns per caption from 6.4 to 20.9 and the share of valid nouns from 8.5% to 13.3%; for SAM images labeled with LLaVA, the figures are 29.3 and 18.6%⁠[30] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. Chen et al., 2024.. SD3 replaced half of its training captions with ones generated by CogVLM, and its overall GenEval score rose from 43.27 to 49.78, with position going from 6.50 to 18.00 and color attribution from 11.75 to 24.75⁠[31] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. Esser et al., 2024..

The same goes for video. CogVideoX first has CogVLM write a dense caption for every frame, then has GPT-4 summarize them into a caption for the whole video, and finally uses the output of this pipeline to train an end-to-end video captioner⁠[32] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. Yang et al., 2025.. ShareGPT4Video has GPT-4V describe, frame by frame, what changes between adjacent frames; it labeled 40K videos this way, trained ShareCaptioner-Video on them, and then used that model to label 4.8M video clips⁠[33] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions. Chen et al., 2024.. Newer models such as SAM 3 can, given a concept, detect and segment every matching object in each frame of a video and track it throughout⁠[34] SAM 3: Segment Anything with Concepts. Carion et al., 2026.; this information is also often used to synthesize more detailed captions.

At inference time, they use a text model to expand the user’s short prompt into a detailed description. The Sora technical report states this clearly: it applies DALL·E 3’s re-captioning to video and uses GPT to expand short user prompts into long captions, which are then sent to the video model⁠[35] Video generation models as world simulators. Brooks et al., 2024.. Movie Gen uses LLaMA3 70B as a teacher to distill an 8B prompt rewriter⁠[36] Movie Gen: A Cast of Media Foundation Models. Polyak et al., 2024.; HunyuanVideo likewise devotes a section to prompt rewriting⁠[37] HunyuanVideo: A Systematic Framework For Large Video Generative Models. Kong et al., 2024.. Prompt Expansion, for its part, found that sampling several expanded versions of the same prompt yields images that are more diverse and also better looking⁠[38] Prompt Expansion for Adaptive Text-to-Image Generation. Datta et al., 2024..

Of course, denser captions are not always better. In controlled experiments, Brack et al. found that long, dense synthetic captions improve text alignment but can come at the cost of aesthetics and sample diversity; randomizing caption length during training eases this trade-off⁠[39] How to Train your Text-to-Image Model: Evaluating Design Choices for Synthetic Training Captions. Brack et al., 2025..

So to a large extent, control over video generation models has become finer and more specific, and has come to depend more and more on the intelligence and understanding of language models.

Vision research teams have also been very active in using synthetic data to keep widening the range that descriptions can cover, a point I’ll come back to when discussing music. Playing for Data rendered 25K images with GTA V, and game data plus one third of CamVid slightly outperformed a model trained on the full CamVid⁠[40] Playing for Data: Ground Truth from Computer Games. Richter et al., 2016.. The teacher in Depth Anything V2 is trained entirely on synthetic images⁠[41] Depth Anything V2. Yang et al., 2024.. CoSyn has an LLM write code to render 400K images and 2.7M rows of instruction data, and its average performance on text-rich benchmarks exceeds that of GPT-4V and Gemini 1.5 Flash⁠[42] Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation. Yang et al., 2025..

Music is doing fine-grained descriptions too, but they’re not fine enough

Using a model to label training data is already routine in music generation as well. LeVo used Qwen2-Audio to give about 2 million songs open-vocabulary tags⁠[43] LeVo: High-Quality Song Generation with Multi-Preference Alignment. Lei et al., 2025.. ACE-Step 1.5 first labeled 5 million samples with Gemini 2.5 Pro as a golden set, then fine-tuned Qwen2.5-Omni into a dedicated captioner, refined it further with a reward model and RL, and finally labeled the full corpus of 27 million samples⁠[44] ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation. Gong et al., 2026.. Captioners themselves are also getting stronger: Music Flamingo’s MF-Skills data covers harmony, structure, timbre, lyrics and cultural context⁠[45] Music Flamingo: Scaling Music Understanding in Audio Language Models. Ghosh et al., 2026., and Qwen has released Qwen3-Omni Captioner as a separate model⁠[46] Qwen3-Omni-30B-A3B-Captioner. Qwen, 2025..

There is work on the time dimension too. FUTGA points out that existing music captioning methods can mostly only write a brief overall description of a short clip, and so built fine-grained captions with temporal information⁠[47] FUTGA: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation. Wu et al., 2024.. SegTune uses Audio Flamingo 3 to write a separate caption for each section, then lays each section’s prompt over the corresponding stretch of time as a condition for generation⁠[48] SegTune: Structured and Fine-Grained Control for Song Generation. Wang et al., 2026..

But these understanding models still fall short in capability. Ideally, a fine-grained description of a piece of music should be detailed enough that the text, written out in full, maps back to one complete piece of audio, just as a sufficiently detailed caption in video maps back to one video. Today’s models are still quite far from this standard. MusTBench ran its tests in 2026 and found that existing audio language models generally struggle even with precise temporal grounding, especially with judging when a sound source ends. Music Flamingo’s average score on temporal sound grounding (TSG) is only 38.5, and only 24.0 for hitting the end point within 3 seconds⁠[49] MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMs. Kwon et al., 2026..

Since video understanding can already do this, we can infer that the shortfall in capability is mainly a shortfall in data. Fine-grained understanding data really is too expensive. MusicCaps has only 5.5k entries, all handwritten by professional musicians⁠[10] MusicLM: Generating Music From Text. Agostinelli et al., 2023.; and it was precisely because manual collection is “costly and time-consuming” that the LP-MusicCaps authors turned to generating pseudo captions from tags with an LLM⁠[50] LP-MusicCaps: LLM-Based Pseudo Music Captioning. Doh et al., 2023..

What if we suddenly had fine-grained understanding?

Suppose a perfect dataset of fine-grained descriptions suddenly fell from the sky, with tens of millions of audio clips, each described in enough detail that the audio could almost be recovered from the text. What would happen? Clearly, music understanding would improve very quickly. These understanding models could be used right away to rewrite the descriptions of the training data for generation models, and the ability of generation models to follow instructions would improve along with them. The three problems I described above would all get better too.

First, the model would no longer get only the gist, and creators could truly control it; it would become a more faithful tool in the creator’s hands. If a description can recover the audio almost perfectly, it is in effect a “project file” that we can edit however we like. In audio-to-audio work, this understanding can also be used to decide what to keep, what not to keep, and how to change things.

Second, instruments would no longer slide away so easily, because there would be less semantic ambiguity. Many tokenizers already use text as supervision: MuCodec adds a lyric recognition loss⁠[13] MuCodec: Ultra Low-Bitrate Music Codec. Xu et al., 2024., and CosyVoice inserts vector quantization into the encoder of a speech recognition model to obtain supervised semantic tokens⁠[51] CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. Du et al., 2024.. If a codec can produce fine-grained text descriptions, that at least shows the information has been preserved; fine-grained text supervision is exactly the kind of supervision signal it needs.

Representation learning has long had related results: without any inductive bias on the model and the data, learning disentangled representations without supervision is theoretically impossible⁠[52] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. Locatello et al., 2019.; and models superimpose more features than they have dimensions onto the same set of dimensions⁠[53] Toy Models of Superposition. Elhage et al., 2022.. There is also an existing framework for measuring how well a representation separates information and how well it retains it, which looks at disentanglement, completeness and informativeness separately⁠[54] A Framework for the Quantitative Evaluation of Disentangled Representations. Eastwood & Williams, 2018..

This benefits the downstream rendering too. Flow matching and diffusion models are actually not very good at handling this kind of ambiguity, and I don’t think it should be left to them. The clearer the condition, the easier the step from semantic to acoustic, and the better the controllability. Seed-Music mentions that because lead sheet tokens don’t capture acoustic features, they needed to scale up the downstream renderer to reach the same end-to-end performance⁠[55] Seed-Music: A Unified Framework for High Quality and Controlled Music Generation. Bai et al., 2024..

Third, the outputs would no longer all sound alike. Once the search space is constrained by the caption, diversity can be achieved by changing the text, and the reward no longer has to be entangled with shortcomings in instruction following. Kaleido Diffusion takes this approach for images: it first autoregressively generates discrete latents such as text descriptions and bounding boxes, then conditions the diffusion on them, trading richer conditions for more diverse outputs⁠[56] Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling. Gu et al., 2024.. This way we can train only the model’s ability to follow instructions, which is easy to verify.

Finally, rewriting, expanding and reasoning over text can all be handed directly to existing LLMs. This lets us draw on the strength of language models: the better they get, the better music generation gets.

This fine-grained description is really a project file

Stem separation, instrument recognition, key, meter, chords, transcription: these traditional MIR problems all come down to how to get a fine-grained caption. Such a dataset won’t fall from the sky, of course, and I think it is most likely to come from DAW project files. These annotations used to be expensive, but the cost is now coming down, because GPT-6 Astra can already control a DAW (Digital Audio Workstation) directly, and can even generate DAW project files outright. In a project, everything is known: which note each track plays at which frame, with what sound, and through what effects. In other words, the labels are frame-level and almost free. Given the pace of the past few years, I believe that through quantization and distillation, intelligence at the same level will only keep getting cheaper.

There is already groundwork for this. Slakh2100 rendered Lakh MIDI with professional sample-based instruments into 2100 multitrack mixes totaling 145 hours⁠[57] Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity. Manilow et al., 2019.. WildFX puts REAPER in Docker and runs it in headless mode, with support for VST, VST3, LV2 and CLAP plugins, for blind estimation of mixing graphs and plugin parameters⁠[58] WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling. Yang et al., 2026.. Systems that let an agent operate a DAW have begun to appear as well: DAWZY has GPT-5 generate Lua scripts that modify a REAPER project through ReaPy, while using three MCP tools to query state, adjust parameters and generate beats⁠[59] DAWZY: A New Addition to AI powered "Human in the Loop" Music Co-creation. Elkins et al., 2025.; the community also has bridges such as AbletonMCP⁠[60] ableton-mcp. Siddharth Ahuja, 2025..

As in vision, the combination of synthetic and real data works in music too. MuScriptor does multi-instrument transcription: trained on synthetic data alone, its onset F1 is 34.5; fine-tuning on real data brings it to 54.4; adding RL brings it to 60.4. The authors conclude that synthetic pretraining is highly effective, especially when real data is scarce, but that synthetic data alone is not enough⁠[61] MuScriptor: An Open Model for Multi-Instrument Music Transcription. Rouard et al., 2026.. In fingerstyle guitar transcription, training on purely procedural data gives an F1 of 66.23, purely real data gives 81.42, and synthetic pretraining followed by real fine-tuning gives 83.49⁠[62] Exploring Procedural Data Generation for Automatic Acoustic Guitar Fingerpicking Transcription. Murgul & Heizmann, 2025.. TAC assembles synthetic mixtures from real audio material to train a timestamped audio captioner and reaches state of the art on TACOS, though the authors also acknowledge that a sim-to-real gap remains⁠[63] TAC: Timestamped Audio Captioning. Kumar et al., 2026..

We have to be honest about this gap between synthetic and real, which comes mainly from the quality of the virtual instruments. SynthSOD trained separation models on synthetic orchestral data, and none of them separated real recordings well, not even the relatively simple URMP. The authors attribute this mainly to the fact that sample-based instruments cycle through a small number of recorded samples in round-robin fashion, so the actual diversity of the data is quite limited⁠[64] SynthSOD: Developing an Heterogeneous Dataset for Orchestra Music Source Separation. Garcia-Martinez et al., 2025..

Suppose we suddenly had virtual instruments of very high quality; we could then have agents generate synthetic data in bulk. Since we know every detail in the project file, the project itself can yield very fine-grained understanding data. Suppose further that agents can arrange like a real person⁠1 They’ll get there. Don’t underestimate what agents can do.; then we would have all the data we need.

With the data, we could first train small classifiers one at a time (stem separation, instrument activity detection, key, meter) and then assemble their outputs into a complete caption. One step beyond that is a transcriber that can transcribe every instrument and recover the complete project directly.

Today we are still a long way from that goal. In the 2025 multi-instrument transcription challenge, only two of the eight teams that submitted valid results beat the MT3 baseline⁠[65] Advancing Multi-Instrument Music Transcription: Results from the 2025 AMT Challenge. Chaturvedi et al., 2025.; systems like YourMT3+ output notes assigned to instruments, that is, pitch and timing, without timbre, effects or mixing parameters yet⁠[66] YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation. Chang et al., 2024..

Using an open DAW project format as the intermediate representation

Besides being used to build understanding data, I think project files can also serve directly as the condition for a generation model. Quite a few works already guide generation with some kind of intermediate condition. Music ControlNet controls generation with time-varying melody, dynamics and rhythm⁠[67] Music ControlNet: Multiple Time-varying Controls for Music Generation. Wu et al., 2024.; JASCO combines symbolic conditions such as chords and melody with audio conditions for temporally controllable generation⁠[68] Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation. Tal et al., 2024.; MuseControlLite supports multiple kinds of control with lightweight conditioners⁠[69] MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners. Tsai et al., 2025.; Seed-Music uses lead sheets as its intermediate representation⁠[55] Seed-Music: A Unified Framework for High Quality and Controlled Music Generation. Bai et al., 2024..

Going further, much recent work first generates a symbolic intermediate representation and then generates the audio: YuE2 first writes out an ABC score⁠[70] YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality. Yuan et al., 2026., StepAudio 3 Music plans the arrangement in ABC notation (ABC-CoT)⁠[71] StepAudio 3 Music Technical Report. Feng et al., 2026., and Qwen-Music plans with melody tokens⁠[72] Qwen-Music Technical Report. Xu et al., 2026..

I think these are all good intermediate steps, but we can go many steps further. If there were an open DAW project format, it would be a better intermediate representation than MIDI or ABC. MIDI basically only covers notes, and ABC records even less; a project file, by contrast, has tracks, clips, notes and expression, automation, plugins and plugin states, so nearly every decision the musician made is recorded in it. Turning it into text and using that as the condition comes very close to having a musician operate the DAW directly. The generation model is then mainly responsible for rendering: the search space is tightly constrained, the distribution it has to learn is more deterministic, and details like instrument timbre are easier to control.

A format like this in fact already exists. DAWproject, introduced by Bitwig and PreSonus, can record the audio, notes, note expression and automation data on the timeline, with plugin states embedded as preset files⁠[73] DAWproject. Bitwig, 2023.. Plugin states use each plugin’s own format, but if every state is converted into a sufficiently detailed description in language, that is no longer a problem.

A shared goal

Putting all this together, the overall goal of AI music as a research field could be to build, as quickly as possible, a large amount of fine-grained text-music paired data, detailed enough that the audio can almost be recovered from the text; and the most feasible technical route at present is to have LLM-driven agents produce synthetic data at scale.

Much of the work mentioned earlier can be seen in light of this goal. Captioners, temporal grounding and transcription go from audio toward description; Slakh2100 and DAW-operating agents go from project toward audio; lead sheets and ABC can be seen as project files that record things rather coarsely. All of them are still some distance from the goal.

Apart from waiting for progress in text models to bring us a cheap, efficient and usable synthetic data agent, there is a lot that can be done. To start the discussion, I list five of them here; each already has some work behind it, and I also try to describe what still separates that work from the goal.

More realistic and more controllable virtual instruments

As mentioned earlier, the gap between synthetic and real data comes mainly from the quality of the virtual instruments, so we need to train a large number of more realistic ones. There is already a good deal of work here. DDSP makes traditional signal processing modules differentiable and trains them together with neural networks, so that pitch and loudness can be controlled independently⁠[74] DDSP: Differentiable Digital Signal Processing. Engel et al., 2020.. MIDI-DDSP builds three levels on top of it, notes, performance and synthesis, aiming both for realistic sound and for detailed user control over the performance⁠[75] MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling. Wu et al., 2022.. torchsynth takes the purely synthetic direction: it is a modular synthesizer that runs on the GPU, released together with one billion sounds and their synthesis parameters⁠[76] One Billion Audio Sounds from GPU-enabled Modular Synthesis. Turian et al., 2021..

People have also trained instruments with methods borrowed from speech synthesis. Deep Performer borrows directly from TTS to synthesize performance audio from a score⁠[77] Deep Performer: Score-to-Audio Music Performance Synthesis. Dong et al., 2022.. Hawthorne et al. have a single model generate spectrograms from MIDI for arbitrary combinations of instruments; they trained one autoregressive and one diffusion version of the decoder, and diffusion was better⁠[78] Multi-instrument Music Synthesis with Spectrogram Diffusion. Hawthorne et al., 2022.. TokenSynth uses a decoder-only transformer to generate codec audio tokens directly from MIDI tokens, with timbre specified by a CLAP embedding⁠[79] TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument. Kim et al., 2025.; MIDI-VALLE, adapted from VALL-E, turns both MIDI and audio into discrete tokens and focuses on piano⁠[80] MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling. Tang et al., 2025.; FlowSynth uses flow matching together with inference-time search to keep an instrument’s timbre consistent across pitches and velocities⁠[81] FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search. Yang et al., 2025..

Measured against the goal, what this work still lacks is mainly control signals and coverage. Its control signal is essentially MIDI, that is, pitch, duration and velocity; how the bow moves, how the player breathes and how each note is attacked are not in MIDI. The instruments covered are also still few: MIDI-VALLE does only piano, and FlowSynth notes that models generating one note at a time have difficulty keeping timbre consistent from note to note. Sample-based instruments, meanwhile, have the problem SynthSOD describes, where the same note is always the same handful of recorded samples over and over⁠[64] SynthSOD: Developing an Heterogeneous Dataset for Orchestra Music Source Separation. Garcia-Martinez et al., 2025..

Perhaps the next things to push on are data collection and the design of control signals. For each instrument, we can start from the player’s perspective, design the control signals it needs, and capture those signals together with the corresponding performance recordings. When data is insufficient, this can be combined with physical modeling, using large amounts of synthetic data for pretraining; or we can train on related instruments first and transfer at the end with a small amount of data from the target instrument. As for the model architecture itself, perhaps we can borrow the approach used in speech and take a small model pretrained on text data to predict the audio token sequence directly from the control token sequence. This could foster collaboration between technical researchers and music students, produce a large number of instruments that music producers can use day to day, and increase the diversity of the synthetic data that agents build.

Making the agent’s arranging, performing and mixing more human

Earlier I assumed that agents can arrange like a real person, which they can’t do yet. There is relatively more work on mixing. Steinmetz et al. built a differentiable mixing console that can learn to mix multitracks from very few examples⁠[82] Automatic multitrack mixing with a differentiable mixing console of neural audio effects. Steinmetz et al., 2021.; Koo et al. trained an encoder that attends only to audio effects and can carry the mixing style of a reference song over to another set of multitracks⁠[83] Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects. Koo et al., 2023.. However, Martínez-Ramírez et al. also write that such models are still far from professional human mixes, one reason being the lack of dry recordings⁠[84] Automatic music mixing with deep learning and out-of-domain data. Martínez-Ramírez et al., 2022.. In performance, piano has data like MAESTRO, with precisely aligned audio and MIDI⁠[85] Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset. Hawthorne et al., 2019., while other instruments have far less. In arrangement, systems like DAWZY can already let an agent edit the project itself⁠[59] DAWZY: A New Addition to AI powered "Human in the Loop" Music Co-creation. Elkins et al., 2025., but I have yet to see work that measures whether its edits resemble those of a real arranger.

Perhaps the first thing to push on is measurement: how to measure the similarity of arranging, performing and mixing decisions. In mixing, the encoder from Koo et al. can count as a start; in arranging and performing, I haven’t seen a counterpart. Once we can measure this, we can study another question: whether, by giving an agent a certain prompt and context (a reference, for example), we can have it imitate the decisions of different arrangers, performers and mixing engineers. As for data, perhaps we can begin by collecting project files of performances and arrangements, since the decisions are all recorded in the project files.

Making rendering into infrastructure everyone can use

Some rendering tools already exist. DawDreamer can host plugins and render audio from Python⁠[86] DawDreamer: Bridging the Gap Between Digital Audio Workstations and Python Interfaces. David Braun, 2021.; WildFX puts REAPER in Docker⁠[58] WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling. Yang et al., 2026.; torchsynth runs more than ten thousand times faster than real time on a single GPU⁠[76] One Billion Audio Sounds from GPU-enabled Modular Synthesis. Turian et al., 2021.. But what is fast in torchsynth is its own synthesizer; good-sounding commercial instruments cost money and are slow to render, and whether the rendered data can be shared publicly remains an open question.

In the short term, perhaps we can purchase commercial instruments to move this process forward while optimizing the infrastructure to make instrument rendering more efficient. As for how to organize this in the long term, perhaps we can draw on what other fields have done and set up organizations so that instrument rendering is carried out in a more distributed and fairer way. Speech has Common Voice, crowdsourced from more than fifty thousand people⁠[87] Common Voice: A Massively-Multilingual Speech Corpus. Ardila et al., 2020., and images have the public LAION-5B⁠[88] LAION-5B: An open large-scale dataset for training next generation image-text models. Schuhmann et al., 2022.; music could certainly have non-profit organizations of its own that regularly share updated datasets through academic publications, so that academic labs and model providers can update their trainers accordingly and offer the corresponding control signals, allowing creators to control the models.

Leave what text can’t describe to inference time

Qualities like timbre and soundstage are hard to describe in language, even with the kind of data described above. For such qualities, we can study how to do a small amount of inference-time search and guidance at the rendering stage, within the given search space, so that music creators can adjust the model’s output with a few samples. There is some related work already: DITTO optimizes the initial noise at inference time to control music generation⁠[89] DITTO: Diffusion Inference-Time T-Optimization for Music Generation. Novack et al., 2024., SMITIN intervenes on MusicGen’s attention heads at inference time⁠[90] SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformers. Koo et al., 2025., and SCORE compares Best-of-N with evolutionary search on general text-to-audio, with the latter improving CLAP and PQ by 10.8% and 10.6% respectively⁠[91] SCORE: Scaling audio generation using Standardized COmposite REwards. Jung et al., 2026..

The search space of these methods is not yet constrained by fine-grained descriptions. Perhaps the next thing to push on is the objective of the search. I haven’t seen work of this kind yet.

Benchmarks that measure how far we are from the goal

Most existing benchmarks don’t measure this goal directly. MuChoMusic tests audio language models’ understanding of music with 1,187 multiple-choice questions and finds that the models rely too heavily on information from the language side⁠[92] MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models. Weck et al., 2024.. CMI-Bench recasts traditional MIR annotations as instructions and finds a large gap between audio language models and specially trained supervised models⁠[93] CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following. Ma et al., 2025.. MusTBench tests temporal grounding⁠[49] MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMs. Kwon et al., 2026.. On the generation side, CMI-RewardBench and SongEval test human preference⁠[25] CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction. Ma et al., 2026.[26] SongEval: A Benchmark Dataset for Song Aesthetics Evaluation. Yao et al., 2025.. None of them directly tests whether a description can recover the audio it describes.

Synthetic data may help here. Because the project file is known, we can have an understanding model describe the rendered audio and then compare its description with the project file, track by track and moment by moment; we can also change one thing in the description and see whether only that thing changes in the generation model’s output. Understanding models and generation models can then be evaluated with the same set of data.

I think these problems are very well suited to academia. They contain many sub-problems, each needing a small loop: pose the problem, collect data, train a model, propose an evaluation. They don’t take up much training compute, but they do require collaboration across disciplines and fields. Solving each small problem moves us toward the larger goal, and each can also be evaluated and used on its own.

Finally

Recently a friend decided to cut off contact with me because I do AI research. She believes AI research is “mutilating people’s souls.” I obviously don’t agree with her, but I can understand why she feels this way; I sometimes feel it myself. When vinyl records and the phonograph were first invented, the orchestras employed by theaters were dismissed in large numbers; but these technologies also gave rise to a new generation of artists, who explored sound effects and non-real-time overdubbing. More importantly, music and language spread, and far more people came to use them.

I believe my friends who make music wouldn’t object to a better Logic Pro X; I also believe my friends who use Ableton can no longer put up with tape-based eight-track recorders. Suno and models like it are not a “better Logic Pro X” today, but they can be. What these fine-grained text descriptions should look like, and what information should go into them, is still something we can decide; together, we can put our own taste and spirit into them.

If we can do that, music models can respect artists’ creative inspiration while letting more people take part, and letting everyone truly benefit from the process. We can also scale this system further and push it to its limit, until it becomes a new kind of tool for making music, a tool truly designed for artists and for artistic creation.

I hope to see this happen; judging from history, I believe it certainly will.

References

  1. Yuan et al. YuE: Scaling Open Foundation Models for Long-Form Music Generation. ICLR, 2026.Section 8.4, human A/B evaluation of 0.5B / 2B / 7B: "Larger models consistently achieve higher preferences"; the 7B model uses 1.75T tokens, the other two 500B. Section 3.2.1: "Accompaniment can overshadow the vocal track, degrading lyric intelligibility."
  2. Copet et al. Simple and Controllable Music Generation. NeurIPS, 2023.Table 5: for 300M / 1.5B / 3.3B, PPL 56.1 / 48.4 / 46.1, FAD 0.96 / 0.86 / 0.82, OVL 78.3 / 81.9 / 79.2; main text: subjective quality stops improving after 1.5B, and 3.3B adheres to the text better
  3. Latent Space. Making Transformers Sing - with Mikey Shulman of Suno. Podcast, 2024."Our models scale similarly to text transformers. It's like bigger is usually better."
  4. Cuervo & Marxer. Scaling Properties of Speech Language Models. EMNLP, 2024."The linguistic performance of SLMs scales up to three orders of magnitude more slowly than that of text-based LLMs."
  5. Zhang et al. Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning. ISMIR, 2025.Instructions support add / remove / extract stem; the instruction data is built from Slakh2100
  6. Suno. Introducing Suno Studio. Suno release notes, 2025.Stem generation; "Send stems out as audio or MIDI"
  7. Schaffer & Singh. RIME: Enabling Large-Scale Agentic Music Post-Production. ICML Workshop on Machine Learning for Audio, 2026.From AL0 instructions (effects and parameters stated directly) to AL2 (only the overall feel described), edit similarity drops: GPT-Audio-Mini 0.718 → 0.519, Gemma 3n 0.687 → 0.528 (arXiv v3; in v1 and v2 the numbers are GPT-4o Mini 0.733 → 0.449, Gemma 3n 0.673 → 0.325)
  8. Liu et al. SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation. ICML, 2025."the model tends to prioritize the more predictable accompaniment, often neglecting the vocal features"
  9. Junyoung Koh. Instrumental Text-to-Music Generation with Auxiliary Conditioning Branches. arXiv, 2026.Saxophone, harmonica and vibraphone "often render as generic timbres"; attributed to data scale (ICME 2026 challenge paper)
  10. Agostinelli et al. MusicLM: Generating Music From Text. arXiv, 2023.Reuses AudioLM's w2v-BERT semantic tokens + SoundStream acoustic tokens; releases MusicCaps (5.5k music-text pairs written by experts)
  11. Li et al. MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training. ICLR, 2024.RVQ-VAE acoustic teacher + CQT musical teacher; "The MFCC features are only good at modelling acoustic and timbre information for single-pitch signals"
  12. Zhu et al. MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization. arXiv, 2025."trained to predict tokens generated by Mel Residual Vector Quantization (Mel-RVQ)"
  13. Xu et al. MuCodec: Ultra Low-Bitrate Music Codec. arXiv, 2024.Stage one: MLM; stage two: mel reconstruction and CQT prediction, plus CTC + RNN-T lyric recognition losses
  14. Chen et al. Rethinking Music Tokenization: A Semantic Codec toward High-Fidelity LLM Music Generation. arXiv, 2026."semantics-guided alternatives are designed for speech and do not fit music well"
  15. Mousavi et al. Discrete Audio Tokens: More Than a Survey! TMLR, 2025."the acoustic-semantic distinction alone is insufficient"; music is "inherently polyphonic and less sparse… highly overlapping sources that are harder to disentangle"
  16. Slendebroek & Metaxa. The Algorithmic Flattening of Sound: Computational Evidence and Justice Implications of AI Music Homogenization. arXiv, 2026.72 MIR features; Lyria 3 loses within-genre diversity, Suno flattens the differences between genres; neither follows the prompt faithfully; attributed to learned priors
  17. Jonason et al. SMART: Tuning a symbolic music generation system with an audio domain aesthetic reward. AIMC, 2025."over-optimization dramatically reduces diversity of model outputs"; after optimization the scores often exceed those of real performances, which the authors treat as a limitation of the proxy metric
  18. Bai et al. DRAGON: Distributional Rewards Optimize Diffusion Generative Models. TMLR, 2025."optimizing per-song FAD worsens Vendi diversity score by an average of 17.7%"
  19. Kirk et al. Understanding the Effects of RLHF on LLM Generalisation and Diversity. ICLR, 2024."RLHF significantly reduces output diversity compared to SFT"
  20. Cideron et al. Diversity-Rewarded CFG Distillation. ICLR, 2025."CFG significantly decreases diversity going from 0.5 (for γ=1) to 0.3 (for γ=7)"; the merged model is more diverse in 57% of comparisons, with quality roughly on par (51%)
  21. GX-Chen et al. KL-Regularized Reinforcement Learning for Generative Modelling is Designed to Mode Collapse. ICLR, 2026.Mode coverage depends mainly on the regularization strength and on the relative scale of rewards and reference probabilities, not on the direction of the KL; low-regularization settings lead to a unimodal distribution
  22. Jiang et al. DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching. arXiv, 2025."As the number of preferences increases, model merging often significantly reduces the average performance"; "improving lyric alignment often compromises musicality"
  23. Kim et al. TuneJury: An Open Metric for Improving Music Generation Preference Alignment. arXiv, 2026.Expert iteration, lr 1e-5: reward +0.416, MAD +0.669 (lower is better); "classic reward-exploitation signature"
  24. Cideron et al. MusicRL: Aligning Music Generation to Human Preferences. ICML, 2024.300,000 pairwise preferences; "text adherence and quality only account for a part of it"; using the quality reward alone "is prone to reward over-optimisation"
  25. Ma et al. CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction. ICML, 2026.500 test samples re-annotated: α = 0.500 (alignment) / 0.504 (musicality); Gemini-2.5-Pro "struggle to exceed 80% agreement"
  26. Yao et al. SongEval: A Benchmark Dataset for Song Aesthetics Evaluation. arXiv, 2025.2,399 full songs, 16 professional annotators, 5 dimensions
  27. Tjandra et al. Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound. arXiv, 2025.Four dimensions: PQ / PC / CE / CU
  28. Xiao et al. On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization. Journal of the American Statistical Association, 2025."preference collapse, where minority preferences are virtually disregarded"
  29. Betker et al. Improving Image Generation with Better Captions. OpenAI, 2023.95% synthetic + 5% original captions; "GPT-4 will readily 'upsample' any caption into a highly descriptive one"
  30. Chen et al. PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR, 2024.Table 1: raw LAION captions have 6.4 nouns per image and 8.5% valid nouns; LAION-LLaVA 20.9 and 13.3%; SAM-LLaVA 29.3 and 18.6%
  31. Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML, 2024.50% original + 50% CogVLM captions; Table 4: GenEval 43.27 → 49.78, position 6.50 → 18.00, color attribution 11.75 → 24.75
  32. Yang et al. CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. ICLR, 2025.CogVLM writes a dense caption for each frame and GPT-4 summarizes them into a video caption; LLaMA 2 is then fine-tuned to replace the summarization step, and the output of the pipeline is used to train the end-to-end CogVLM2-Caption
  33. Chen et al. ShareGPT4Video: Improving Video Understanding and Generation with Better Captions. NeurIPS, 2024.Differential sliding-window captioning (DiffSW); 40K videos annotated by GPT-4V; ShareCaptioner-Video annotates 4.8M videos
  34. Carion et al. SAM 3: Segment Anything with Concepts. ICLR, 2026."detects, segments, and tracks objects in images and videos based on concept prompts"
  35. Brooks et al. Video generation models as world simulators. OpenAI, 2024."We apply the re-captioning technique introduced in DALL·E 3 to videos"; "leverage GPT to turn short user prompts into longer detailed captions"
  36. Polyak et al. Movie Gen: A Cast of Media Foundation Models. arXiv, 2024.Fine-tuned 8B and 70B LLaMA3 video captioners annotate the whole training set; the 70B teacher is distilled into an 8B prompt rewriter
  37. Kong et al. HunyuanVideo: A Systematic Framework For Large Video Generative Models. arXiv, 2024.Section 4.6, Prompt Rewrite
  38. Datta et al. Prompt Expansion for Adaptive Text-to-Image Generation. ACL, 2024.Multiple expanded prompts produce "more aesthetically pleasing and diverse" images
  39. Brack et al. How to Train your Text-to-Image Model: Evaluating Design Choices for Synthetic Training Captions. ICCV Workshops, 2025.Dense captions improve alignment but may cost aesthetics and diversity; random-length captions "without compromising sample diversity"
  40. Richter et al. Playing for Data: Ground Truth from Computer Games. ECCV, 2016.25K images from GTA V; game data + 1/3 of CamVid outperforms the full CamVid
  41. Yang et al. Depth Anything V2. NeurIPS, 2024.teacher "purely on high-quality synthetic images"
  42. Yang et al. Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation. ACL, 2025.400K images and 2.7M rows of instruction data; "surpass proprietary models such as GPT-4V and Gemini 1.5 Flash"
  43. Lei et al. LeVo: High-Quality Song Generation with Multi-Preference Alignment. NeurIPS, 2025."annotate all tracks utilizing Qwen2-Audio to obtain open-vocabulary tags"; about 2 million songs
  44. Gong et al. ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation. arXiv, 2026.Gemini 2.5 Pro annotates a golden set of 5M samples; Qwen2.5-Omni is fine-tuned into a captioner and refined with RL; all 27M samples are annotated
  45. Ghosh et al. Music Flamingo: Scaling Music Understanding in Audio Language Models. ICLR, 2026.MF-Skills covers "harmony, structure, timbre, lyrics, and cultural context"
  46. Qwen. Qwen3-Omni-30B-A3B-Captioner. Hugging Face, 2025.A dedicated audio captioner model
  47. Wu et al. FUTGA: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation. NLP4MusA, 2024."Existing music captioning methods are limited to generating concise global descriptions of short music clips"
  48. Wang et al. SegTune: Structured and Fine-Grained Control for Song Generation. ACL, 2026.Audio Flamingo 3 captions each section; section prompts are broadcast over their time windows as conditions
  49. Kwon et al. MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMs. arXiv, 2026."existing LALMs struggle with precise temporal grounding"; Music Flamingo averages 38.5 on TSG, offset Hit@3s 24.0
  50. Doh et al. LP-MusicCaps: LLM-Based Pseudo Music Captioning. ISMIR, 2023."costly and time-consuming collection process"; generates pseudo captions from tags with an LLM
  51. Du et al. CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens. arXiv, 2024."supervised semantic tokens… derived from a multilingual speech recognition model by inserting vector quantization into the encoder"
  52. Locatello et al. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. ICML, 2019.Without inductive biases on both the models and the data, unsupervised learning of disentangled representations is fundamentally impossible
  53. Elhage et al. Toy Models of Superposition. Transformer Circuits Thread, 2022.Models superimpose more features than they have dimensions onto the same set of dimensions
  54. Eastwood & Williams. A Framework for the Quantitative Evaluation of Disentangled Representations. ICLR, 2018.The DCI framework: disentanglement, completeness and informativeness are measured separately
  55. Bai et al. Seed-Music: A Unified Framework for High Quality and Controlled Music Generation. arXiv, 2024."As the lead sheet tokens lack acoustic feature characterization, we need to scale up the token-to-latent diffusion model"
  56. Gu et al. Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling. NeurIPS, 2024.Autoregressively generates latents such as captions and bounding boxes to "diversify and enrich the input conditions"
  57. Manilow et al. Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity. WASPAA, 2019.2100 tracks, 145 hours, rendered from Lakh MIDI with professional sample-based instruments
  58. Yang et al. WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling. DAFx, 2026.Runs REAPER headless in Docker; VST / VST3 / LV2 / CLAP; blind estimation of mixing graphs and plugin parameters
  59. Elkins et al. DAWZY: A New Addition to AI powered "Human in the Loop" Music Co-creation. NeurIPS AI for Music Workshop, 2025."GPT-5 generates Lua that ReaPy executes to modify the project"; three MCP tools
  60. Siddharth Ahuja. ableton-mcp. GitHub, 2025.A community-built MCP bridge for Ableton
  61. Rouard et al. MuScriptor: An Open Model for Multi-Instrument Music Transcription. ISMIR, 2026.Onset F1 (α_CFG = 2): synthetic only 34.5, with real-data fine-tuning 54.4, with RL on top 60.4; synthetic pretraining is "highly effective, especially when only little real music audio data is available", but on its own "not sufficient"
  62. Murgul & Heizmann. Exploring Procedural Data Generation for Automatic Acoustic Guitar Fingerpicking Transcription. AIMC, 2025.Table 6, F1: procedural 66.23, real 81.42, procedural + fine-tuning 83.49
  63. Kumar et al. TAC: Timestamped Audio Captioning. arXiv, 2026.Trains on synthetic mixtures built from real material; state of the art on TACOS; acknowledges a sim-to-real gap
  64. Garcia-Martinez et al. SynthSOD: Developing an Heterogeneous Dataset for Orchestra Music Source Separation. IEEE Open Journal of Signal Processing, 2025."none of the models were able to achieved strong separation results even in URMP"; the authors attribute this mainly to round-robin sampling limiting the actual diversity of the data
  65. Chaturvedi et al. Advancing Multi-Instrument Music Transcription: Results from the 2025 AMT Challenge. NeurIPS AI for Music Workshop, 2025."Eight teams submitted valid solutions; two outperformed the baseline MT3 model."
  66. Chang et al. YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation. MLSP, 2024.Outputs notes assigned to instruments (pitch and timing)
  67. Wu et al. Music ControlNet: Multiple Time-varying Controls for Music Generation. TASLP, 2024.Time-varying melody, dynamics and rhythm controls
  68. Tal et al. Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation. ISMIR, 2024.Temporally controlled text-to-music generation with joint symbolic and audio conditioning
  69. Tsai et al. MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners. ICML, 2025.Multiple kinds of control with lightweight conditioners
  70. Yuan et al. YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality. arXiv, 2026.First writes an editable ABC score (tempo, meter, key, chords, vocal and instrumental melodies), then generates semantic tokens and acoustic latents conditioned on it
  71. Feng et al. StepAudio 3 Music Technical Report. arXiv, 2026.ABC-CoT: first writes an arrangement plan in ABC notation (chords, tempo, meter, key, bars, melody), then predicts music tokens conditioned on it
  72. Xu et al. Qwen-Music Technical Report. arXiv, 2026.Melody-CoT: first generates discrete melody tokens (vocal pitch, expressed in relative MIDI semitones), then semantic tokens; the plan uses melody tokens, not ABC
  73. Bitwig. DAWproject. GitHub, 2023.Encodes audio, notes, note expressions and automation; "Plug-in states (always embedded)"; list of supported DAWs
  74. Engel et al. DDSP: Differentiable Digital Signal Processing. ICLR, 2020.Makes traditional signal processing modules differentiable and trains them together with neural networks; pitch and loudness can be controlled independently
  75. Wu et al. MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling. ICLR, 2022.An instrument model with three levels: notes, performance and synthesis; "enables both realistic neural audio synthesis and detailed user control"
  76. Turian et al. One Billion Audio Sounds from GPU-enabled Modular Synthesis. ISMIR, 2021.synth1B1: one billion 4-second synthesized sounds, all with their synthesis parameters; torchsynth runs 16,200 times faster than real time on a single GPU
  77. Dong et al. Deep Performer: Score-to-Audio Music Performance Synthesis. ICASSP, 2022."we borrow recent advances in text-to-speech synthesis"; adapts the transformer encoder-decoder for polyphony and long notes
  78. Hawthorne et al. Multi-instrument Music Synthesis with Spectrogram Diffusion. ISMIR, 2022.MIDI to spectrogram to audio, for arbitrary combinations of instruments; the decoder was trained once autoregressively and once as a DDPM, and "the DDPM approach is superior"
  79. Kim et al. TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument. ICASSP, 2025."a decoder-only transformer to generate desired audio tokens from MIDI tokens and CLAP … embedding"; instrument cloning and timbre specified by text
  80. Tang et al. MIDI-VALLE: Improving Expressive Piano Performance Synthesis Through Neural Codec Language Modelling. ISMIR, 2025.Adapted from VALL-E; MIDI and audio are both encoded as discrete tokens; piano only; FAD more than 75% lower than the baseline on ATEPP and Maestro
  81. Yang et al. FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search. arXiv, 2025."maintaining consistent timbre across different pitches and velocities, a challenge that existing note-level models struggle to address"; flow matching with test-time search, outperforming TokenSynth
  82. Steinmetz et al. Automatic multitrack mixing with a differentiable mixing console of neural audio effects. ICASSP, 2021."Applications of deep learning to automatic multitrack mixing are largely unexplored. This is partly due to the limited available data"
  83. Koo et al. Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects. ICASSP, 2023.An encoder pretrained with contrastive learning extracts only the information related to audio effects from a reference song, and the mixing style of the input multitrack is converted to that of the reference
  84. Martínez-Ramírez et al. Automatic music mixing with deep learning and out-of-domain data. ISMIR, 2022."the lack of dry or clean instrument recordings limits the performance of such models, which is still far from professional human-made mixes"
  85. Hawthorne et al. Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset. ICLR, 2019.Piano performances with precisely aligned audio and MIDI; Wave2Midi2Wave, with notes as the intermediate representation
  86. David Braun. DawDreamer: Bridging the Gap Between Digital Audio Workstations and Python Interfaces. ISMIR Late-Breaking Demo, 2021.A DAW framework that hosts plugins and renders audio from Python
  87. Ardila et al. Common Voice: A Massively-Multilingual Speech Corpus. LREC, 2020.Data collection and validation both rely on crowdsourcing; "Over 50,000 individuals have participated so far"
  88. Schuhmann et al. LAION-5B: An open large-scale dataset for training next generation image-text models. NeurIPS Datasets and Benchmarks, 2022.An open large-scale dataset of image-text pairs
  89. Novack et al. DITTO: Diffusion Inference-Time T-Optimization for Music Generation. ICML, 2024."controlling pre-trained text-to-music diffusion models at inference-time via optimizing initial noise latents"
  90. Koo et al. SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformers. IEEE Open Journal of Signal Processing, 2025.Intervenes on MusicGen's attention heads along probe directions at inference time
  91. Jung et al. SCORE: Scaling audio generation using Standardized COmposite REwards. IEEE Signal Processing Letters, 2026.Evolutionary search (EvoSearch) "improves CLAP by 10.8% and PQ by 10.6%", against 9.2% and 8.4% for Best-of-N; a single reward causes "verifier hacking"; the experiments are on general text-to-audio
  92. Weck et al. MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models. ISMIR, 2024.1,187 multiple-choice questions on 644 tracks, all validated by humans; finds an "over-reliance on the language modality" in the models
  93. Ma et al. CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following. ISMIR, 2025.Rewrites traditional MIR annotations in instruction-following form; "significant performance gaps between LLMs and supervised models"

Citation

Please cite this post as:

Yongyi Zang. "Music generation models need finer-grained text descriptions". yongyi.dev, Oct 2026. https://yongyi.dev/en/blog/fine-grained-captions-for-music-generation/

Or use the BibTeX entry:

@misc{zang2026finegrainedcaptions,
  author = {Yongyi Zang},
  title  = {Music generation models need finer-grained text descriptions},
  year   = {2026},
  month  = {oct},
  url    = {https://yongyi.dev/en/blog/fine-grained-captions-for-music-generation/},
}