Zero-shot Cross-lingual Voice Transfer for TTS, and Its Application to Voice Restoration for Accessibility and Inclusion
We present a zero-shot voice transfer (VT) module that can be easily plugged into a state-of-the-art TTS system to restore the voices of input speakers.
We include the following audio samples for our research blogpost.
Zero-shot Examples using Typical Reference Speech
Below are zero-shot examples using typical reference speech, to simulate the scenario when the speaker's voice was recorded before any voice degradation occurred. we demonstrate the concept of zero-shot capability using samples from the VCTK corpus:
Reference
TTS with Zero-shot VT
Female (P257)
Male (P256)
Female (P244)
Male (P243)
Female (P253)
Male (P285)
Female (P228)
Male (P241)
Female (P236)
Male (P360)
Female (P248)
Cross-lingual Zero-shot Examples using Typical Reference Speech
We also evaluate the cross-lingual capability of our TTS zero-shot model on typical reference speech across six different languages, using English reference speakers from the VCTK corpus. The transcripts and their translations were automatically generated using Gemini:
Reference
English
Spanish
French
German
Arabic
Chinese Mandarin
Japanese
Male (P246)
Female (P303)
Male (P256)
Female (P244)
Male (P243)
Female (P231)
Male (P285)
Female (P282)
Speaker Similarity
85% ± 6.3%
76% ± 3.6%
62% ± 12.0%
75% ± 6.9%
70% ± 5.5%
74% ± 11.1%
70% ± 14.4%
70% ± 5.5%
MOS Naturalness
3.300 ± .29
3.621 ± .05
3.606 ± .06
4.058 ± .04
3.985 ± .04
4.242 ± .03
3.921 ± .05
3.616 ± .05
Case study: Atypical Speech as a Reference
We work with two Googlers Dimitri Kanevsky and Aubrie Lee to synthesize the videos below using only 12 seconds of Dimitri's atypical voice and 14 seconds of Aubrie's atypical voice as reference, respectively.
Participants
Original Videos
Videos with VT Outputs
Dimitri Kanevsky
Aubrie Lee
We also test whether our model that makes use of the same atypical English reference speech from Dimitri and Aubrie can generalize and transfer their voice to other languages, given non-English text.
Below are VT outputs in six different languages for Dimitri (French, Spanish, Italian, Arabic, German, Russian):
French
Spanish
Italian
Arabic
German
Russian
Below are VT outputs in six different languages for Aubrie (French, Spanish, Italian, Arabic, Hindi, Norwegian):