mirror of
https://github.com/rhasspy/piper-sample-generator.git
synced 2026-08-27 18:15:58 -04:00
Update README
This commit is contained in:
Michael Hansen
parent
4057c1a620
commit
4d7e4b390c
3 changed files with 155 additions and 123 deletions
101
README.md
101
README.md
|
|
@ -1,14 +1,8 @@
|
|||
# Piper Sample Generator
|
||||
|
||||
Generates samples using [Piper](https://github.com/rhasspy/piper/) for training a wake word system like [openWakeWord](https://github.com/dscripka/openWakeWord).
|
||||
|
||||
Available models:
|
||||
|
||||
* [English](https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/en_US-libritts_r-medium.pt)
|
||||
* [French](https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/fr_FR-mls-medium.pt)
|
||||
* [German](https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/de_DE-mls-medium.pt)
|
||||
* [Dutch](https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/nl_NL-mls-medium.pt)
|
||||
Generate spoken audio samples using [Piper][piper] for training a wake word system like [openWakeWord][] or [microWakeWord][].
|
||||
|
||||
Supports normal [Piper voices][piper voices] or a special [generator][] that can mix speaker embeddings (English only).
|
||||
|
||||
## Install
|
||||
|
||||
|
|
@ -21,23 +15,45 @@ cd piper-sample-generator/
|
|||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
python3 -m pip install --upgrade pip
|
||||
python3 -m pip install -r requirements.txt
|
||||
python3 -m pip install -e .
|
||||
```
|
||||
|
||||
Download the LibriTTS-R generator (exported from [checkpoint](https://huggingface.co/datasets/rhasspy/piper-checkpoints/tree/main/en/en_US/libritts_r/medium)):
|
||||
## Piper Voices
|
||||
|
||||
Download one or more [Piper voices][piper voices] (both the `.onnx` and `.onnx.json` files for each voice). [Audio samples][piper samples] are available.
|
||||
|
||||
As an example, we'll download the U.S. English "lessac" voice in medium quality:
|
||||
|
||||
``` sh
|
||||
mkdir -p voices
|
||||
wget -O voices/en_US-lessac-medium.onnx 'https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx?download=true'
|
||||
wget -O voices/en_US-lessac-medium.onnx.json 'https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json?download=true'
|
||||
```
|
||||
|
||||
Generate a small set of samples with the CLI:
|
||||
|
||||
``` sh
|
||||
python3 generate_samples.py 'okay piper.' --model voices/en_US-lessac-medium.onnx --max-samples 10 --output-dir okay_piper/
|
||||
```
|
||||
|
||||
Check the `okay_piper/` directory for 10 WAV files (named `0.wav` to `9.wav`).
|
||||
|
||||
You can add multiple `--model <voice>` arguments to cycle between different voices when generating samples.
|
||||
|
||||
See `--help` for more options, including `--length-scales` (speaking speeds).
|
||||
|
||||
## Generator
|
||||
|
||||
Download the LibriTTS-R generator (exported from [checkpoint][]):
|
||||
|
||||
``` sh
|
||||
wget -O models/en-us-libritts-high.pt 'https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/en_US-libritts_r-medium.pt'
|
||||
```
|
||||
|
||||
See links above for models for other languages.
|
||||
|
||||
## Run
|
||||
|
||||
Generate a small set of samples with the CLI:
|
||||
|
||||
``` sh
|
||||
python3 generate_samples.py 'okay, piper.' --max-samples 10 --output-dir okay_piper/
|
||||
python3 generate_samples.py 'okay piper.' --model models/en-us-libritts-high.pt --max-samples 10 --output-dir okay_piper/
|
||||
```
|
||||
|
||||
Check the `okay_piper/` directory for 10 WAV files (named `0.wav` to `9.wav`).
|
||||
|
|
@ -45,63 +61,38 @@ Check the `okay_piper/` directory for 10 WAV files (named `0.wav` to `9.wav`).
|
|||
Generation can be much faster and more efficient if you have a GPU available and PyTorch is configured to use it. In this case, increase the batch size:
|
||||
|
||||
``` sh
|
||||
python3 generate_samples.py 'okay, piper.' --max-samples 100 --batch-size 10 --output-dir okay_piper/
|
||||
python3 generate_samples.py 'okay piper.' --model models/en-us-libritts-high.pt --max-samples 100 --batch-size 10 --output-dir okay_piper/
|
||||
```
|
||||
|
||||
On an NVidia 2080 Ti with 11GB, a batch size of 100 was possible (generating approximately 100 samples per second).
|
||||
|
||||
Setting `--max-speakers` to a value less than 904 (the number of speakers LibriTTS) is recommended. Because very few samples of later speakers were in the original dataset, using them can cause audio artifacts.
|
||||
|
||||
See `--help` for more options, including adjust the `--length-scales` (speaking speeds) and `--slerp-weights` (speaker blending) which are cycled per batch.
|
||||
|
||||
Alternatively, you can import the generate function into another Python script:
|
||||
|
||||
```python
|
||||
from generate_samples import generate_samples # make sure to add this to your Python path as needed
|
||||
|
||||
generate_samples(text = ["okay, piper"], max_samples = 100, output_dir = output_dir, batch_size=10)
|
||||
```
|
||||
|
||||
There are some additional arguments available when importing the function directly, see the docstring of `generate_sample` for more information.
|
||||
See `--help` for more options, including the `--length-scales` (speaking speeds) and `--slerp-weights` (speaker blending) which are cycled per batch.
|
||||
|
||||
### Augmentation
|
||||
|
||||
Once you have samples generating, you can augment them using [audiomentation](https://iver56.github.io/audiomentations/):
|
||||
Once you have samples generated, you can augment them using [audiomentation](https://iver56.github.io/audiomentations/):
|
||||
|
||||
``` sh
|
||||
python3 augment.py --sample-rate 16000 okay_piper/ okay_piper_augmented/
|
||||
python3 augment.py --sample-rate 22050 okay_piper/ okay_piper_augmented/
|
||||
```
|
||||
|
||||
This will do several things to each sample:
|
||||
|
||||
1. Randomly decrease the volume
|
||||
* The original samples are normalized, so different volume levels are needed
|
||||
2. Randomly [apply an impulse response](https://iver56.github.io/audiomentations/waveform_transforms/apply_impulse_response/) using the files in `impulses/`
|
||||
2. Randomly apply an [impulse response][] using the files in `impulses/`
|
||||
* Change the acoustics of the sample to sound like the speaker was in a room with echo or using a poor quality microphone
|
||||
3. Resample to 16Khz for training (e.g., [openWakeWord](https://github.com/dscripka/openWakeWord))
|
||||
3. Resample to 16Khz for training (e.g., [openWakeWord][])
|
||||
|
||||
|
||||
## Short Phrases
|
||||
|
||||
Models that were trained on audio books tend to perform poorly when speaking short phrases or single words.
|
||||
The French, German, and Dutch models trained from the [MLS](http://openslr.org/94/) have this problem.
|
||||
|
||||
The problem can be mitigated by repeating the phrase over and over, and then clipping out a single sample.
|
||||
To do this automatically, follow these steps:
|
||||
|
||||
1. Ensure your short phrase ends with a comma (`<phrase>,`)
|
||||
2. Lower the noise settings with `--noise-scales 0.333` and `--noise-scale-ws 0.333`
|
||||
3. Use `--min-phoneme-count 300` (the value 300 was determined empirically and may be less for some models)
|
||||
|
||||
For example:
|
||||
|
||||
``` sh
|
||||
python3 generate_samples.py \
|
||||
'framboise,' \
|
||||
--model models/fr_FR-mls-medium.pt \
|
||||
--noise-scales 0.333 \
|
||||
--noise-scale-ws 0.333 \
|
||||
--min-phoneme-count 300
|
||||
--max-samples 1 \
|
||||
--output-dir .
|
||||
```
|
||||
<!-- Links -->
|
||||
[piper]: https://github.com/OHF-Voice/piper1-gpl/
|
||||
[openWakeWord]: https://github.com/dscripka/openWakeWord
|
||||
[microWakeWord]: https://github.com/kahrendt/microWakeWord/
|
||||
[piper voices]: https://huggingface.co/rhasspy/piper-voices
|
||||
[generator]: https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/en_US-libritts_r-medium.pt
|
||||
[piper samples]: https://rhasspy.github.io/piper-samples/
|
||||
[checkpoint]: https://huggingface.co/datasets/rhasspy/piper-checkpoints/tree/main/en/en_US/libritts_r/medium
|
||||
[impulse response]: https://iver56.github.io/audiomentations/waveform_transforms/apply_impulse_response/
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ def main() -> None:
|
|||
|
||||
augment = Compose(
|
||||
transforms=[
|
||||
Gain(min_gain_in_db=-12, max_gain_in_db=0),
|
||||
Gain(min_gain_db=-12, max_gain_db=0),
|
||||
ApplyImpulseResponse(impulses),
|
||||
]
|
||||
)
|
||||
|
|
|
|||
|
|
@ -26,9 +26,9 @@ logging.basicConfig(level=logging.DEBUG)
|
|||
def generate_samples(
|
||||
text: Union[List[str], str],
|
||||
output_dir: Union[str, Path],
|
||||
model: Union[str, Path],
|
||||
max_samples: Optional[int] = None,
|
||||
file_names: Optional[Iterable[str]] = None,
|
||||
model: Union[str, Path] = _DIR / "models" / "en_US-libritts_r-medium.pt",
|
||||
batch_size: int = 1,
|
||||
slerp_weights: Tuple[float, ...] = (0.5,),
|
||||
length_scales: Tuple[float, ...] = (0.75, 1, 1.25),
|
||||
|
|
@ -45,10 +45,10 @@ def generate_samples(
|
|||
text (List[str]): The text to convert into speech. Can be either a
|
||||
a list of strings, or a path to a file with text on each line.
|
||||
output_dir (str): The location to save the generated clips.
|
||||
model (str): The path to the TTS generator model (.pt).
|
||||
max_samples (int): The maximum number of samples to generate.
|
||||
file_names (List[str]): The names to use when saving the files. Must be the same length
|
||||
as the `text` argument, if a list.
|
||||
model (str): The path to the TTS generator model (.pt).
|
||||
batch_size (int): The batch size to use when generated the clips
|
||||
slerp_weights (List[float]): The weights to use when mixing speakers via SLERP.
|
||||
length_scales (List[float]): Controls the average duration/speed of the generated speech.
|
||||
|
|
@ -100,17 +100,6 @@ def generate_samples(
|
|||
)
|
||||
)
|
||||
|
||||
# Define resampler to get to 16khz (https://pytorch.org/audio/stable/tutorials/audio_resampling_tutorial.html#kaiser-best)
|
||||
resample_rate = 16000
|
||||
resampler = torchaudio.transforms.Resample(
|
||||
sample_rate,
|
||||
resample_rate,
|
||||
lowpass_filter_width=64,
|
||||
rolloff=0.9475937167399596,
|
||||
# resampling_method="sinc_interp_kaiser",
|
||||
beta=14.769656459379492,
|
||||
)
|
||||
|
||||
speakers_iter = it.cycle(it.product(range(num_speakers), range(num_speakers)))
|
||||
speakers_batch = list(it.islice(speakers_iter, 0, batch_size))
|
||||
if isinstance(text, str) and os.path.exists(text):
|
||||
|
|
@ -157,22 +146,23 @@ def generate_samples(
|
|||
return padded_lists
|
||||
|
||||
phoneme_ids_by_batch = right_pad_lists(phoneme_ids_by_batch)
|
||||
audio = generate_audio(
|
||||
torch_model,
|
||||
speaker_1,
|
||||
speaker_2,
|
||||
phoneme_ids_by_batch,
|
||||
slerp_weight,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
max_len,
|
||||
audio = (
|
||||
generate_audio(
|
||||
torch_model,
|
||||
speaker_1,
|
||||
speaker_2,
|
||||
phoneme_ids_by_batch,
|
||||
slerp_weight,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
max_len,
|
||||
)
|
||||
.cpu()
|
||||
.numpy()
|
||||
)
|
||||
|
||||
# Resample audio
|
||||
audio_np = resampler(audio.cpu()).numpy()
|
||||
|
||||
audio_int16 = audio_float_to_int16(audio_np)
|
||||
audio_int16 = audio_float_to_int16(audio)
|
||||
for audio_idx in range(audio_int16.shape[0]):
|
||||
audio_data = np.trim_zeros(audio_int16[audio_idx].flatten())
|
||||
|
||||
|
|
@ -183,7 +173,7 @@ def generate_samples(
|
|||
|
||||
wav_file: wave.Wave_write = wave.open(str(wav_path), "wb")
|
||||
with wav_file:
|
||||
wav_file.setframerate(resample_rate)
|
||||
wav_file.setframerate(sample_rate)
|
||||
wav_file.setsampwidth(2)
|
||||
wav_file.setnchannels(1)
|
||||
wav_file.writeframes(audio_data)
|
||||
|
|
@ -209,7 +199,7 @@ def generate_samples(
|
|||
def generate_samples_onnx(
|
||||
text: Union[List[str], str],
|
||||
output_dir: Union[str, Path],
|
||||
model: Union[str, Path],
|
||||
model: Union[str, Path, List[Union[str, Path]]],
|
||||
max_samples: Optional[int] = None,
|
||||
file_names: Optional[Iterable[str]] = None,
|
||||
length_scales: Tuple[float, ...] = (0.75, 1, 1.25),
|
||||
|
|
@ -241,21 +231,20 @@ def generate_samples_onnx(
|
|||
if max_samples is None:
|
||||
max_samples = len(text)
|
||||
|
||||
if not isinstance(model, list):
|
||||
model = [model]
|
||||
|
||||
_LOGGER.debug("Loading %s", model)
|
||||
voice = PiperVoice.load(model, use_cuda=torch.cuda.is_available())
|
||||
_LOGGER.info("Successfully loaded the model")
|
||||
voices = [PiperVoice.load(m, use_cuda=torch.cuda.is_available()) for m in model]
|
||||
_LOGGER.info("Successfully loaded model(s)")
|
||||
|
||||
output_dir = Path(output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
num_speakers = voice.config.num_speakers
|
||||
if max_speakers is not None:
|
||||
num_speakers = min(num_speakers, max_speakers)
|
||||
|
||||
sample_idx = 0
|
||||
settings_iter = it.cycle(
|
||||
it.product(
|
||||
list(range(num_speakers)),
|
||||
voices,
|
||||
length_scales,
|
||||
noise_scales,
|
||||
noise_scale_ws,
|
||||
|
|
@ -278,27 +267,32 @@ def generate_samples_onnx(
|
|||
if file_names:
|
||||
file_names = it.cycle(file_names)
|
||||
|
||||
for speaker_id, length_scale, noise_scale, noise_w_scale in settings_iter:
|
||||
if isinstance(file_names, it.cycle):
|
||||
wav_path = output_dir / next(file_names)
|
||||
else:
|
||||
wav_path = output_dir / f"{sample_idx}.wav"
|
||||
for voice, length_scale, noise_scale, noise_w_scale in settings_iter:
|
||||
num_speakers = voice.config.num_speakers
|
||||
if max_speakers is not None:
|
||||
num_speakers = min(num_speakers, max_speakers)
|
||||
|
||||
wav_file: wave.Wave_write = wave.open(str(wav_path), "wb")
|
||||
voice.synthesize_wav(
|
||||
next(texts),
|
||||
wav_file=wav_file,
|
||||
syn_config=SynthesisConfig(
|
||||
speaker_id=speaker_id,
|
||||
length_scale=length_scale,
|
||||
noise_scale=noise_scale,
|
||||
noise_w_scale=noise_w_scale,
|
||||
),
|
||||
)
|
||||
for speaker_id in range(num_speakers):
|
||||
if isinstance(file_names, it.cycle):
|
||||
wav_path = output_dir / next(file_names)
|
||||
else:
|
||||
wav_path = output_dir / f"{sample_idx}.wav"
|
||||
|
||||
sample_idx += 1
|
||||
if sample_idx >= max_samples:
|
||||
break
|
||||
wav_file: wave.Wave_write = wave.open(str(wav_path), "wb")
|
||||
voice.synthesize_wav(
|
||||
next(texts),
|
||||
wav_file=wav_file,
|
||||
syn_config=SynthesisConfig(
|
||||
speaker_id=speaker_id,
|
||||
length_scale=length_scale,
|
||||
noise_scale=noise_scale,
|
||||
noise_w_scale=noise_w_scale,
|
||||
),
|
||||
)
|
||||
|
||||
sample_idx += 1
|
||||
if sample_idx >= max_samples:
|
||||
return
|
||||
|
||||
_LOGGER.info("Done")
|
||||
|
||||
|
|
@ -459,45 +453,92 @@ def audio_float_to_int16(
|
|||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main() -> None:
|
||||
def main() -> int:
|
||||
"""Main entry point."""
|
||||
|
||||
# Get command line arguments
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("text")
|
||||
parser.add_argument("--max-samples", required=True, type=int)
|
||||
parser.add_argument(
|
||||
"--model", default=_DIR / "models" / "en_US-libritts_r-medium.pt"
|
||||
"--max-samples",
|
||||
required=True,
|
||||
type=int,
|
||||
help="Maximum number of samples to generate",
|
||||
)
|
||||
parser.add_argument("--batch-size", type=int, default=1)
|
||||
parser.add_argument("--slerp-weights", nargs="+", type=float, default=[0.5])
|
||||
parser.add_argument(
|
||||
"--length-scales", nargs="+", type=float, default=[1.0, 0.75, 1.25, 1.4]
|
||||
"--model",
|
||||
required=True,
|
||||
action="append",
|
||||
help="Path to PyTorch generator (.pt) or Piper voice model (.onnx)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch-size", type=int, default=1, help="CUDA batch size (generator only)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--slerp-weights",
|
||||
nargs="+",
|
||||
type=float,
|
||||
default=[0.5],
|
||||
help="Speaker blending weights (generator only)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--length-scales",
|
||||
nargs="+",
|
||||
type=float,
|
||||
default=[1.0, 0.75, 1.25, 1.4],
|
||||
help="Audio length scales (< 1 is faster, > 1 is slower)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--noise-scales",
|
||||
nargs="+",
|
||||
type=float,
|
||||
default=[0.667, 0.75, 0.85, 0.9, 1.0, 1.4],
|
||||
help="Noise amounts added to audio (most voices use 0.667)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--noise-scale-ws",
|
||||
nargs="+",
|
||||
type=float,
|
||||
default=[0.8],
|
||||
help="Phoneme width variation (most voices use 0.8)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
default="output",
|
||||
help="Directory to output WAV files (default: ./output)",
|
||||
)
|
||||
parser.add_argument("--noise-scale-ws", nargs="+", type=float, default=[0.8])
|
||||
parser.add_argument("--output-dir", default="output")
|
||||
parser.add_argument(
|
||||
"--max-speakers",
|
||||
type=int,
|
||||
help="Maximum number of speakers to use (default: all)",
|
||||
help="Maximum number of speakers to use (default: no limit)",
|
||||
)
|
||||
parser.add_argument("--verbose", action="store_true")
|
||||
args = parser.parse_args().__dict__
|
||||
|
||||
# Generate speech
|
||||
model_path = Path(args["model"])
|
||||
if model_path.suffix == ".onnx":
|
||||
model_paths = [Path(m) for m in args["model"]]
|
||||
assert model_paths
|
||||
|
||||
if any(mp for mp in model_paths[1:] if mp.suffix != model_paths[0].suffix):
|
||||
_LOGGER.error("All models must have the same suffix (.pt or .onnx)")
|
||||
return 1
|
||||
|
||||
if model_paths[0].suffix == ".onnx":
|
||||
# Use Piper voice (.onnx)
|
||||
generate_samples_onnx(**args)
|
||||
else:
|
||||
elif model_paths[0].suffix == ".pt":
|
||||
# Use PyTorch generator (.pt)
|
||||
if len(model_paths) > 1:
|
||||
_LOGGER.error("Only one generator (.pt) is supported")
|
||||
return 1
|
||||
|
||||
args["model"] = args["model"][0]
|
||||
generate_samples(**args)
|
||||
else:
|
||||
_LOGGER.error("Models must have .pt or .onnx suffix")
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue