Improving Text-To-Audio Models with Synthetic Captions

It is an open challenge to obtain high quality training data, especially captions, for text-to-audio models. Although prior methods have leveraged $\mathit{\text{text-only}}\mathit{\text{language}}\mathit{\text{models}}$ to augment and improve captions, such methods have limitations related to scale and coherence between audio and captions. In this work, we propose an audio captioning pipeline that uses an $\mathit{\text{audio}}\mathit{\text{language}}\mathit{\text{model}}$ to synthesize accurate and diverse captions for audio at scale. We leverage this pipeline to produce a dataset of synthetic captions for AudioSet, named $\mathtt{\text{AF-AudioSet}}$, and then evaluate the benefit of pre-training text-to-audio models on these synthetic captions. Through systematic evaluations on AudioCaps and MusicCaps, we find leveraging our pipeline and synthetic captions leads to significant improvements on audio generation quality, achieving a new $\mathit{\text{state-of-the-art}}$.