Adapting TTS models For New Speakers using Transfer Learning

The original TTS model is trained on speaker 8051 on more than 27 hours of speech and text pairs. The spectrogram synthesis model is a FastPitch model with a learnable alignment module, while the vocoder model is a HiFiGAN vocoder. The real validation examples along with the synthesized examples for the validation text are presented in the table below.

In this finetuning approach, we finetune all the parameters of the pre-trained TTS models directly on the data of the new speaker. For finetuning the spectrogram-synthesis model, we require the text and speech pairs of the new speaker, while for the vocoder model, we only require the speech examples of the speaker for generating the spectrogram and waveform pairs. The real validation examples along with the synthesized examples from the validation text are presented in the table below.

Direct finetuning can result in overfitting or catastrophic forgetting when the amount of training data of the new speaker is very limited. To address this challenge, we explore another transfer learning method in which we mix the original speaker's data with the new speaker's data during finetuning. In this setting, we assume that we have enough training samples of the original speaker while the number of samples of the new speaker is limited. We create a data-loading pipeline that samples equal number of examples from the original and new speaker in each mini-batch. We add a speaker embedding layer in the spectrogram synthesis module, to make it suitable for multiple speaker training. The vocoder model architecture is unchanged and the model is finetuned on the spectrogram and audio pairs of the two speakers. The real validation examples along with the synthesized examples from the validation text are presented in the table below.