Authors: Wei-Ning Hsu, Yu Zhang, Ron J. Weiss, Yu-An Chung, Yuxuan Wang, Yonghui Wu, James Glass
Abstract:
To leverage crowd-sourced data to train multi-speaker text-to-speech (TTS) models that can synthesize clean speech for all speakers, it is essential to learn disentangled representations which can independently control the speaker identity and background noise in generated signals. However, learning such representations can be challenging, due to the lack of labels describing the recording conditions of each training example, and the fact that speakers and recording conditions are often correlated, e.g. since users often make many recordings using the same equipment. This paper proposes three components to address this problem by: (1) formulating a conditional generative model with factorized latent variables, (2) using data augmentation to add noise that is not correlated with speaker identity and whose label is known during training, and (3) using adversarial factorization to improve disentanglement. Experimental results demonstrate that the proposed method can disentangle speaker and noise attributes even if they are correlated in the training data, and can be used to consistently synthesize clean speech for all speakers. Ablation studies verify the importance of each proposed component.
In this section, we present reference audio used to infer latent speaker variables zs and latent residual variables zr. These latent variables are used to control the speaker identity and noise condition, respectively.
This section contains synthesized audio from the baseline model and the proposed model using Griffin-Lim algorithm. The baseline model utilizes a speaker embedding table to control speaker identity. The proposed model uses zs to control speaker identity and zr to control the acoustic condition. These examples demonstrate model's ability to synthesize clean speech for all speakers, regardless of the training data quality of the speaker. In contrast, the baseline model always generates noisy speech for speakers whose training data are noisy.
Text 1: Try these pages.
Text 2: In 2009 these reports were collected in the book Chambermaids and Soldiers.
Text 3: In later years in films she switched to playing character parts.
Text 4: The other bodies are juxtaposed in various unlit areas behind them.
In this section, we compare using Griffin-Lim and WaveRNN to synthesize waveforms using spectrograms generated by the proposed model. The results demonstrate that using WaveRNN can improve the audio fidelity.
Text: The Godfather Release date is March 15, 1972
(zs(1), zr(1))
(zs(2), zr(1))
(zs(3), zr(1))
(zs(4), zr(1))
Griffin-Lim
WaveRNN
Text: The Palladium-Item is the daily morning newspaper for Richmond, Indiana and surrounding areas.