98 lines
3.6 KiB
Markdown
98 lines
3.6 KiB
Markdown
|
# Wavenet
|
||
|
|
|
||
|
|
Paddle implementation of wavenet in dynamic graph, a convolutional network based vocoder. Wavenet is proposed in [WaveNet: A Generative Model for Raw Audio](https://arxiv.org/abs/1609.03499), but in thie experiment, the implementation follows the teacher model in [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](arxiv.org/abs/1807.07281).
|
||
|
|
|
||
|
|
|
||
|
|
## Dataset
|
||
|
|
|
||
|
|
We experiment with the LJSpeech dataset. Download and unzip [LJSpeech](https://keithito.com/LJ-Speech-Dataset/).
|
||
|
|
|
||
|
|
```bash
|
||
|
|
wget https://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2
|
||
|
|
tar xjvf LJSpeech-1.1.tar.bz2
|
||
|
|
```
|
||
|
|
|
||
|
|
## Project Structure
|
||
|
|
|
||
|
|
```text
|
||
|
|
├── data.py data_processing
|
||
|
|
├── configs/ (example) configuration file
|
||
|
|
├── synthesis.py script to synthesize waveform from mel_spectrogram
|
||
|
|
├── train.py script to train a model
|
||
|
|
└── utils.py utility functions
|
||
|
|
```
|
||
|
|
|
||
|
|
## Train
|
||
|
|
|
||
|
|
Train the model using train.py, follow the usage displayed by `python train.py --help`.
|
||
|
|
|
||
|
|
```text
|
||
|
|
usage: train.py [-h] [--data DATA] [--config CONFIG] [--output OUTPUT]
|
||
|
|
[--device DEVICE] [--resume RESUME]
|
||
|
|
|
||
|
|
Train a wavenet model with LJSpeech.
|
||
|
|
|
||
|
|
optional arguments:
|
||
|
|
-h, --help show this help message and exit
|
||
|
|
--data DATA path of the LJspeech dataset.
|
||
|
|
--config CONFIG path of the config file.
|
||
|
|
--output OUTPUT path to save results.
|
||
|
|
--device DEVICE device to use.
|
||
|
|
--resume RESUME checkpoint to resume from.
|
||
|
|
```
|
||
|
|
|
||
|
|
1. `--config` is the configuration file to use. The provided configurations can be used directly. And you can change some values in the configuration file and train the model with a different config.
|
||
|
|
2. `--data` is the path of the LJSpeech dataset, the extracted folder from the downloaded archive (the folder which contains metadata.txt).
|
||
|
|
3. `--resume` is the path of the checkpoint. If it is provided, the model would load the checkpoint before trainig.
|
||
|
|
4. `--output` is the directory to save results, all result are saved in this directory. The structure of the output directory is shown below.
|
||
|
|
|
||
|
|
```text
|
||
|
|
├── checkpoints # checkpoint
|
||
|
|
└── log # tensorboard log
|
||
|
|
```
|
||
|
|
|
||
|
|
5. `--device` is the device (gpu id) to use for training. `-1` means CPU.
|
||
|
|
|
||
|
|
example script:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python train.py --config=./configs/wavenet_single_gaussian.yaml --data=./LJSpeech-1.1/ --output=experiment --device=0
|
||
|
|
```
|
||
|
|
|
||
|
|
You can monitor training log via tensorboard, using the script below.
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd experiment/log
|
||
|
|
tensorboard --logdir=.
|
||
|
|
```
|
||
|
|
|
||
|
|
## Synthesis
|
||
|
|
```text
|
||
|
|
usage: synthesis.py [-h] [--data DATA] [--config CONFIG] [--device DEVICE]
|
||
|
|
checkpoint output
|
||
|
|
|
||
|
|
Synthesize valid data from LJspeech with a wavenet model.
|
||
|
|
|
||
|
|
positional arguments:
|
||
|
|
checkpoint checkpoint to load.
|
||
|
|
output path to save results.
|
||
|
|
|
||
|
|
optional arguments:
|
||
|
|
-h, --help show this help message and exit
|
||
|
|
--data DATA path of the LJspeech dataset.
|
||
|
|
--config CONFIG path of the config file.
|
||
|
|
--device DEVICE device to use.
|
||
|
|
```
|
||
|
|
|
||
|
|
1. `--config` is the configuration file to use. You should use the same configuration with which you train you model.
|
||
|
|
2. `--data` is the path of the LJspeech dataset. A dataset is not needed for synthesis, but since the input is mel spectrogram, we need to get mel spectrogram from audio files.
|
||
|
|
3. `checkpoint` is the checkpoint to load.
|
||
|
|
4. `output_path` is the directory to save results. The output path contains the generated audio files (`*.wav`).
|
||
|
|
5. `--device` is the device (gpu id) to use for training. `-1` means CPU.
|
||
|
|
|
||
|
|
example script:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python synthesis.py --config=./configs/wavenet_single_gaussian.yaml --data=./LJSpeech-1.1/ --device=0 experiment/checkpoints/step_500000 generated
|
||
|
|
```
|