mirror of
https://github.com/dscripka/openWakeWord.git
synced 2026-08-27 18:17:20 -04:00
398 lines
15 KiB
Text
398 lines
15 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f49227ca",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Introduction"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6483fd4d",
|
|
"metadata": {},
|
|
"source": [
|
|
"This notebook demonstrates how to train custom openWakeWord models using pre-defined datasets, an automated process for synthetic data generation/augmentation, and a custom training process. While not guaranteed to always produce the best performing model, the methods shown in this notebook often produce baseline models with relatively strong performance.\n",
|
|
"\n",
|
|
"Manual data preparation and model training (e.g., see the [training models](training_models.ipynb) notebook) remains an option for when full control over the model development process is needed.\n",
|
|
"\n",
|
|
"At a high level, the automatic training process takes advantages of several techniques to try and produce a good model, including:\n",
|
|
"\n",
|
|
"- Early-stopping and checkpoint averaging (similar to [stochastic weight averaging](https://arxiv.org/abs/1803.05407)) to search for the best models found during training, according to the validation data\n",
|
|
"- Variable learning rates with cosine decay and multiple cycles\n",
|
|
"- Adaptive batch construction to focus on only high-loss examples when the model begins to converge, combined with gradient accumulation to ensure that batch sizes are still large enough for stable training\n",
|
|
"- Cycical weight schedules for negative examples to help the model reduce false-positive rates\n",
|
|
"\n",
|
|
"See the contents of the `train.py` file for more details."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "67ea460d",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Environment Setup"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "004ef4db",
|
|
"metadata": {},
|
|
"source": [
|
|
"To begin, we'll need to install the requirements for training custom models. In particular, a relatively recent version of Pytorch and custom fork of the [piper-sample-generator](https://github.com/dscripka/piper-sample-generator) library for generating synthetic examples for the custom model.\n",
|
|
"\n",
|
|
"**Important Note!** Currently, automated model training is only supported on linux systems due to the requirements of the text to speech library used for synthetic sample generation (Piper). It may be possible to use Piper on Windows/Mac systems, but that has not (yet) been tested."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ad77a769",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"## Environment setup\n",
|
|
"\n",
|
|
"# install piper-sample-generator (currently only supports linux systems)\n",
|
|
"!git clone https://github.com/dscripka/piper-sample-generator\n",
|
|
"!wget -O piper-sample-generator/models/en-us-libritts-high.pt 'https://github.com/rhasspy/piper-sample-generator/releases/download/v1.0.0/en-us-libritts-high.pt'\n",
|
|
"!apt-get install libespeak-ng1 # may not be required on all systems\n",
|
|
"!pip install espeak_phonemizer\n",
|
|
"\n",
|
|
"# install openwakeword (full installation to support training)\n",
|
|
"!git clone https://github.com/dscripka/openwakeword\n",
|
|
"!pip install -e ./openwakeword[full]\n",
|
|
"!cd openwakeword\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "da85818d",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-09-04T13:42:01.183840Z",
|
|
"start_time": "2023-09-04T13:41:59.752153Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Imports\n",
|
|
"\n",
|
|
"import os\n",
|
|
"import torch\n",
|
|
"from openwakeword.data import mmap_batch_generator, generate_adversarial_texts\n",
|
|
"from openwakeword.utils import compute_features_from_generator\n",
|
|
"import sys\n",
|
|
"from pathlib import Path\n",
|
|
"import uuid\n",
|
|
"import yaml\n",
|
|
"import datasets\n",
|
|
"import scipy\n",
|
|
"from tqdm import tqdm\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "658ccb2a",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Download Data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1e4450ab",
|
|
"metadata": {},
|
|
"source": [
|
|
"When training new openWakeWord models using the automated procedure, four specific types of data are required:\n",
|
|
"\n",
|
|
"1) Synthetic examples of the target word/phrase generated with text-to-speech models\n",
|
|
"\n",
|
|
"2) Synthetic examples of adversarial words/phrases generated with text-to-speech models\n",
|
|
"\n",
|
|
"3) Room impulse reponses and noise/background audio data to augment the synthetic examples and make them more realistic\n",
|
|
"\n",
|
|
"4) Generic \"negative\" audio data that is very unlikely to contain examples of the target word/phrase in the context where the model should detect it. This data can be the original audio data, or precomputed openWakeWord features ready for model training.\n",
|
|
"\n",
|
|
"5) Validation data to use for early-stopping when training the model.\n",
|
|
"\n",
|
|
"For the purposes of this notebook, all five of these sources will either be generated manually or can be obtained from HuggingFace thanks to their excellent `datasets` library and extremely generous hosting policy. Also note that while only a portion of some datasets are downloaded, for the best possible performance it is recommended to download the entire dataset and keep a local copy for future training runs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ab3c1fce",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-09-04T01:07:17.746749Z",
|
|
"start_time": "2023-09-04T01:07:17.740846Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Download room impulse responses collected by MIT\n",
|
|
"# https://mcdermottlab.mit.edu/Reverb/IR_Survey.html\n",
|
|
"\n",
|
|
"output_dir = \"./mit_rirs\"\n",
|
|
"if not os.path.exists(output_dir):\n",
|
|
" os.mkdir(output_dir)\n",
|
|
"rir_dataset = datasets.load_dataset(\"davidscripka/MIT_environmental_impulse_responses\", split=\"train\", streaming=True)\n",
|
|
"\n",
|
|
"for row in tqdm(rir_dataset):\n",
|
|
" name = row['audio']['path'].split('/')[-1]\n",
|
|
" scipy.io.wavfile.write(os.path.join(output_dir, name), 16000, row['audio']['array'])\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b28717ef",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"## Download noise and background audio\n",
|
|
"\n",
|
|
"# FSD50k Noise Dataset (warning, this can take 5 minutes to prepare when streaming)\n",
|
|
"# https://zenodo.org/record/4060432\n",
|
|
"\n",
|
|
"output_dir = \"./fsd50k\"\n",
|
|
"if not os.path.exists(output_dir):\n",
|
|
" os.mkdir(output_dir)\n",
|
|
"fsd50k_dataset = datasets.load_dataset(\"Fhrozen/FSD50k\", split=\"validation\", streaming=True) # ~40,000 files in this split\n",
|
|
"fsd50k_dataset = iter(fsd50k_dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16000)))\n",
|
|
"\n",
|
|
"n_total = 500 # use only 500 clips for this example notebook, recommend increasing for full-scale training\n",
|
|
"for i in tqdm(range(n_total)):\n",
|
|
" row = next(fsd50k_dataset)\n",
|
|
" name = row['audio']['path'].split('/')[-1]\n",
|
|
" scipy.io.wavfile.write(os.path.join(output_dir, name), 16000, row['audio']['array'])\n",
|
|
" i += 1\n",
|
|
" if i == n_total:\n",
|
|
" break\n",
|
|
"\n",
|
|
"# Free Music Archive dataset\n",
|
|
"# https://github.com/mdeff/fma\n",
|
|
"\n",
|
|
"output_dir = \"./fma\"\n",
|
|
"if not os.path.exists(output_dir):\n",
|
|
" os.mkdir(output_dir)\n",
|
|
"fma_dataset = datasets.load_dataset(\"rudraml/fma\", name=\"small\", split=\"train\", streaming=True)\n",
|
|
"fma_dataset = iter(fma_dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16000)))\n",
|
|
"\n",
|
|
"n_hours = 1 # use only 1 hour of clips for this example notebook, recommend increasing for full-scale training\n",
|
|
"for i in tqdm(range(n_hours*3600//30)): # this works because the FMA dataset is all 30 second clips\n",
|
|
" row = next(fma_dataset)\n",
|
|
" name = row['audio']['path'].split('/')[-1]\n",
|
|
" scipy.io.wavfile.write(os.path.join(output_dir, name), 16000, row['audio']['array'])\n",
|
|
" i += 1\n",
|
|
" if i == n_hours*3600//30:\n",
|
|
" break\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b5bbe225",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Download pre-computed openWakeWord features for training and validation\n",
|
|
"\n",
|
|
"# training set (~2,000 hours)\n",
|
|
"!wget https://huggingface.co/datasets/davidscripka/openwakeword_features/blob/main/openwakeword_features_ACAV100M_2000_hrs_16bit.npy\n",
|
|
"\n",
|
|
"# validation set for false positive rate estimation (~11 hours)\n",
|
|
"!wget https://huggingface.co/datasets/davidscripka/openwakeword_features/blob/main/validation_set_features.npy"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9265dbcf",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Define Training Configuration"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1e204f3e",
|
|
"metadata": {},
|
|
"source": [
|
|
"For automated model training openWakeWord uses a specially designed training script and a [YAML](https://yaml.org/) configuration file that defines all of the information required for training a new wake word/phrase detection model.\n",
|
|
"\n",
|
|
"It is strongly recommended that you review [the example config file](../examples/custom_model.yml), as each value is fully documented there. For the purposes of this notebook, we'll read in the YAML file to modify certain configuration parameters before saving a new YAML file for training our example model. Specifically:\n",
|
|
"\n",
|
|
"- We'll train a detection model for the phrase \"hey sebastian\"\n",
|
|
"- We'll only generate 5,000 positive and negative examples (to save on time for this example)\n",
|
|
"- We'll only generate 1,000 validation positive and negative examples for early stopping (again to save time)\n",
|
|
"- The model will only be trained for 10,000 steps (larger datasets will benefit from longer training)\n",
|
|
"- We'll reduce the target metrics to account for the small dataset size and limited training.\n",
|
|
"\n",
|
|
"On the topic of target metrics, there are *not* specific guidelines about what these metrics should be in practice, and you will need to conduct testing in your target deployment environment to establish good thresholds. However, from very limited testing the default values in the config file (accuracy >= 0.7, recall >= 0.5, false-positive rate <= 0.2 per hour) seem to produce model with reasonable performance.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c0fea7ab",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-09-04T18:11:33.893397Z",
|
|
"start_time": "2023-09-04T18:11:33.878938Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Load default YAML config file for training\n",
|
|
"config = yaml.load(open(\"openwakeword/examples/custom_model.yml\", 'r').read(), yaml.Loader)\n",
|
|
"config"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "fbe862c5",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-09-04T15:07:00.859210Z",
|
|
"start_time": "2023-09-04T15:07:00.841472Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Modify values in the config and save a new version\n",
|
|
"\n",
|
|
"config[\"target_phrase\"] = [\"hey sebastian\"]\n",
|
|
"config[\"n_samples\"] = 1000\n",
|
|
"config[\"n_samples_val\"] = 1000\n",
|
|
"config[\"steps\"] = 10000\n",
|
|
"config[\"target_accuracy\"] = 0.6\n",
|
|
"config[\"target_recall\"] = 0.25\n",
|
|
"\n",
|
|
"config[\"background_paths\"] = []#['./fsd50k', './fma']\n",
|
|
"config[\"false_positive_validation_data_path\"] = \"val_set_features.npy\"\n",
|
|
"config[\"feature_data_files\"] = {\"ACAV100M_sample\": \"openwakeword_features_ACAV100M_2000_hrs_16bit.npy\"}\n",
|
|
"\n",
|
|
"with open('my_model.yaml', 'w') as file:\n",
|
|
" documents = yaml.dump(config, file)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c7460b02",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Train the Model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c1de9710",
|
|
"metadata": {},
|
|
"source": [
|
|
"With the data downloaded and training configuration set, we can now start training the model. We'll do this in parts to better illustrate the sequence, but you can also execute every step at once for a fully automated process."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "85115fc9",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-09-04T13:50:08.803326Z",
|
|
"start_time": "2023-09-04T13:50:06.790241Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Step 1: Generate synthetic clips\n",
|
|
"# For the number of clips we are using, this should take ~10 minutes on a free Google Colab instance\n",
|
|
"# If generation fails, you can simply run this command again as it will continue generating until the\n",
|
|
"# number of files meets the targets specified in the config file\n",
|
|
"\n",
|
|
"!{sys.executable} openwakeword/openwakeword/train.py --training_config my_model.yaml --generate_clips"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9efa15af",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-09-04T13:56:08.781018Z",
|
|
"start_time": "2023-09-04T13:55:40.203515Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Step 2: Augment the generated clips\n",
|
|
"\n",
|
|
"!{sys.executable} openwakeword/openwakeword/train.py --training_config my_model.yaml --augment_clips"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "916b511a",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-09-04T15:11:14.742260Z",
|
|
"start_time": "2023-09-04T15:07:03.755159Z"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Step 3: Train model\n",
|
|
"\n",
|
|
"!{sys.executable} openwakeword/openwakeword/train.py --training_config my_model.yaml --train_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4925229c",
|
|
"metadata": {},
|
|
"source": [
|
|
"After the model finishes training, the auto training script will automatically convert it to ONNX and tflite versions, saving them as `<model_name>.onnx/tflite` in the present working directory, where `<model_name>` is defined in the YAML training config file.\n",
|
|
"\n",
|
|
"At this point, you can take the trained models and use them as normal with openWakeWord."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "openwakeword_dev",
|
|
"language": "python",
|
|
"name": "openwakeword_dev"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.16"
|
|
},
|
|
"toc": {
|
|
"base_numbering": 1,
|
|
"nav_menu": {},
|
|
"number_sections": true,
|
|
"sideBar": true,
|
|
"skip_h1_title": false,
|
|
"title_cell": "Table of Contents",
|
|
"title_sidebar": "Contents",
|
|
"toc_cell": false,
|
|
"toc_position": {},
|
|
"toc_section_display": true,
|
|
"toc_window_display": false
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|