Audio samples from "Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model"

Paper: arXiv
Authors: Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, Rif A. Saurous
Abstract: A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module. Building these components often requires extensive domain expertise and may contain brittle design choices. In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters. Given <text, audio> pairs, the model can be trained completely from scratch with random initialization. We present several key techniques to make the sequence-to-sequence framework perform well for this challenging task. Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness. In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.

All of the below phrases are unseen by Tacotron during training.

Tacotron works well on out-of-domain and complex words.

“Generative adversarial network or variational auto-encoder.”
“Basilar membrane and otolaryngology are not auto-correlations.”

Tacotron learns pronunciations based on phrase semantics.

(Note how Tacotron pronounces "read" in the two phrases.)
“He has read the whole thing.”
“He reads books.”

Tacotron is somewhat robust to spelling errors.

“Thisss isrealy awhsome.”

Tacotron is sensitive to punctuation.

(Note how the comma in the first phrase changes prosody.)
“This is your personal assistant, Google Home.”
“This is your personal assistant Google Home.”

Tacotron learns stress and intonation.

(The speaker is instructed to stress on capitalized words in our training set. So simply capitalizing some words will change the overall intonation.)
“The buses aren't the problem, they actually provide a solution.”
“The buses aren't the PROBLEM, they actually provide a SOLUTION.”

Tacotron's prosody changes when turning a statement into a question.

“The quick brown fox jumps over the lazy dog.”
“Does the quick brown fox jump over the lazy dog?”

Ablation Experiments

Comparing vanilla seq2seq vs. Tacotron

“Talib Kweli confirmed to AllHipHop that he will be releasing an album in the next year.”
Vanilla seq2seq
Tacotron

Comparing the GRU encoder vs. the CBHG encoder

“The blue lagoon is a nineteen eighty American romance adventure film.”
GRU encoder
CBHG encoder
“Talib Kweli confirmed to AllHipHop that he will be releasing an album in the next year.”
GRU encoder
CBHG encoder

Comparing with and without the post-processing network.

“Tajima Airport serves Toyooka.”
Without
With