No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS

Recent work reports gains in neural text-to-speech (TTS) with Group Relative Policy Optimization (GRPO). However, in the absence of a verifiable reward for $\mathit{\text{prosody}}$, GRPO trained on transcription-oriented signals (CER/NLL) lowers error rates yet collapses prosody into monotone, unnatural speech; adding speaker-similarity further destabilizes training and degrades CER. We address this with an $\mathit{\text{iterative}}\mathit{\text{Direct}}\mathit{\text{Preference}}\mathit{\text{Optimization}}\mathit{\text{(DPO)}}$ scheme that uses only a few hundred human-labeled preference pairs per round to directly optimize prosodic naturalness while regularizing to the current model. On $\mathbf{\text{KoCC-TTS}}$, a curated dataset of authentic Korean call center interactions capturing task-oriented dialogues, our method attains the highest human preference (ELO) with competitive CER, outperforming GRPO and strong commercial baselines. These results suggest that when prosody cannot be rewarded automatically, $\mathit{\text{human}}\mathit{\text{preference}}\mathit{\text{optimization}}$ offers a practical and data-efficient path to natural and robust TTS. The demo page is available at \href{https://tts.ch.dev}