FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency

Text-based speech editing (TSE) allows users to edit speech by modifying the corresponding text directly without altering the original recording. Current TSE techniques often focus on minimizing discrepancies between generated speech and reference within edited regions during training to achieve fluent TSE performance. However, the generated speech in the edited region should maintain acoustic and prosodic consistency with the unedited region and the original speech at both the local and global levels. To maintain speech fluency, we propose a new fluency speech editing scheme based on our previous $\mathit{\text{FluentEditor}}$ model, termed $\mathit{\text{FluentEditor2}}$, by modeling the multi-scale acoustic and prosody consistency training criterion in TSE training. Specifically, for local acoustic consistency, we propose $\mathit{\text{hierarchical}}\mathit{\text{local}}\mathit{\text{acoustic}}\mathit{\text{smoothness}}\mathit{\text{constraint}}$ to align the acoustic properties of speech frames, phonemes, and words at the boundary between the generated speech in the edited region and the speech in the unedited region. For global prosody consistency, we propose $\mathit{\text{contrastive}}\mathit{\text{global}}\mathit{\text{prosody}}\mathit{\text{consistency}}\mathit{\text{constraint}}$ to keep the speech in the edited region consistent with the prosody of the original utterance. Extensive experiments on the VCTK and LibriTTS datasets show that $\mathit{\text{FluentEditor2}}$ surpasses existing neural networks-based TSE methods, including Editspeech, Campnet, A${}^3$T, FluentSpeech, and our Fluenteditor, in both subjective and objective. Ablation studies further highlight the contributions of each module to the overall effectiveness of the system. Speech demos are available at: \url{https://github.com/Ai-S2-Lab/FluentEditor2}.