Abstract:
With the growth of computing power on mobile phones and privacy concerns over user’s data, on-device real time speech pro-cessing has become an important research topic. In this paper, we focus on methods for real time spectrogram inversion, where an algorithm receives a portion of the input signal (e.g., one frame) and processes it incrementally, i.e., operating in streaming mode. We present a real time Griffin Lim(GL) algorithm using a sliding window approach in STFT domain. The proposed algorithm is 2.4x faster than real time on the ARM CPU of a Pixel4. In addition we explore a neural vocoder operating in streaming mode and demonstrate the impact of looking ahead on perceptual quality. As little as one hop size (12.5ms) of lookahead is able to significantly improve perceptual quality in comparison to a causal model. We compare GL with the neural vocoder and show different trade-offs in terms of perceptual quality, on-device latency, algorithmic delay, memory footprint and noise sensitivity. For fair quality assessment of the GL approach, we use input log magnitude spectrogram without mel transformation. We evaluate presented real time spectrogram inversion approaches on clean, noisy and atypical speech.
In this section, we compare the ground truth audio file, together with non-streaming GL with 70 iterations (NonStreamGLi70), nonstreaming GL with 3 iterations (NonStreamGLi3), streaming GL (StreamGLs4i4c2), streaming neural vocoder with one hop lookahead (StreamMelGANlookahead1), non-streaming neural vocoder (NonStreamMelGANlookahead), and the causal vocoder (StreamMelGANcausal0).
Parrotron is an end-to-end speech conversion model that is trained to convert atypical speech to fluent speech of a canonical synthesized voice. This model takes a log-mel spectrogram as an input and directly produces linear spectrogram. It requires a vocoder to synthesize a time-domain waveform in real time, thus a lightweight streaming vocoder is a crucial requirement for such an application.
In this section, we present examples of running Parrotron with different vocoders to convert atypical speech from a deaf speaker to fluent speech. We compare the ground truth audio file, together with non-streaming GL with 70 iterations (NonStreamGLi70), streaming GL (StreamGLs4i4c2), and streaming neural vocoder with one hop lookahead (StreamMelGANlookahead1).