akadmin/openwakeword
1
0
Fork 0
mirror of https://github.com/dscripka/openWakeWord.git synced 2026-08-27 18:17:20 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #48 from dscripka/auto_training

Browse source 
Auto training
This commit is contained in:
dscripka 2023-10-11 21:49:39 -04:00 • committed by GitHub
commit acb12ed061
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 1698 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

20
README.md
View file

@ -210,7 +210,13 @@ While the models are trained with background noise to increase robustness, in so
# Training New Models
Training new models is conceptually simple, and the entire process is demonstrated in a [tutorial notebook](notebooks/training_models.ipynb).
openWakeWord includes an automated utility that greatly simplifies the process of training custom models. This can be used in two ways:
1) In a simple [Google Colab](https://colab.research.google.com/drive/1q1oe2zOyZp7UsB3jJiQ1IFn8z5YfjwEb?usp=sharing) notebook with an easy to use interface and simple end-to-end process. This allows anyone to produce a custom model very quickly (<1 hour) and doesn't require any development experience, but the performance of the model may be low in some deployment scenarios.
2) A more detailed [notebook](notebooks/automatic_model_training.ipynb) (also on [Google Colab](https://colab.research.google.com/drive/1yyFH-fpguX2BTAW8wSQxTrJnJTM-0QAd?usp=sharing)) that describes the training process in more details, and enables more customization. This can produce high quality models, but requires more development experience.
For users interested in understanding the fundamental concepts behind model training there is a more detailed, educational [tutorial notebook](notebooks/training_models.ipynb) also available. However, this specific notebook is not intended for training production models, and the automated process above is recommended for that purpose.
Fundamentally, a new model requires two data generation and collection steps:
@ -233,7 +239,7 @@ Future release road maps may have non-english support. In particular, [Mycroft.A
- While the ONNX runtime [does support javascript](https://onnxruntime.ai/docs/get-started/with-javascript.html), much of the other functionality required for openWakeWord models would need to be ported. This is not currently on the roadmap, but please open an issue/start a discussion if this feature is of particular interest.
**Is there a C++ version of openWakeWord?**
- While the ONNX runtime [also has a C++ API](https://onnxruntime.ai/docs/get-started/with-cpp.html), there isn't an official C++ implementation of the full openWakeWord library. However, [@synesthesiam](https://github.com/synesthesiam) has created a [C++ version](https://github.com/rhasspy/openWakeWord-cpp) of openWakeWord with the essential functionality implemented.
- While the ONNX runtime [also has a C++ API](https://onnxruntime.ai/docs/get-started/with-cpp.html), there isn't an official C++ implementation of the full openWakeWord library. However, [@synesthesiam](https://github.com/synesthesiam) has created a [C++ version](https://github.com/rhasspy/openWakeWord-cpp) of openWakeWord with basic functionality implemented.
**Why are there three separate models instead of just one?**
- Separating the models was an intentional choice to provide flexibility and optimize the efficiency of the end-to-end prediction process. For example, with separate melspectrogram, embedding, and prediction models, each one can operate on different size inputs of audio to optimize overall latency and share computations between models. It certainly is possible to make a combined model with all of the steps integrated, though, if that was a requirement of a particular use case.
@ -241,6 +247,16 @@ Future release road maps may have non-english support. In particular, [Mycroft.A
**I still get a large number of false activations when I use the pre-trained models, how can I reduce these?**
- First, review the [recommendations for usage](#recommendations-for-usage) and ensure that these options do not improve overall system accuracy. Second, experiment with [custom verifier models](#user-specific-models), if possible. If neither of these approaches are helping, please open an issue with details of the deployment environment and the types of false activations that you are experiencing. We certainly appreciate feedback & requests on how to improve the base pre-trained models!
# Acknowledgements
I am very grateful for the encouraging and positive response from the open-source community since the release of openWakeWord in January 2023. In particular, I want to acknowledge and thank the following individuals and groups for their feedback, collaboration, and development support:
- [synesthesiam](https://github.com/synesthesiam)
- [SecretSauceAI](https://github.com/secretsauceai)
- [OpenVoiceOS](https://github.com/OpenVoiceOS)
- [Nabu Casa](https://github.com/NabuCasa)
- [Home Assistant](https://github.com/home-assistant)
# License
All of the code in this repository is licensed under the **Apache 2.0** license. All of the included pre-trained models are licensed under the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/) license due to the inclusion of datasets with unknown or restrictive licensing as part of the training data. If you are interested in pre-trained models with more permissive licensing, please raise an issue and we will try to add them to a future release.

101
examples/custom_model.yml Normal file
View file

@ -0,0 +1,101 @@
## Configuration file to be used with `train.py` to create custom wake word/phrase models
# The name of the model (will be used when creating directoires and when saving the final .onnx and .tflite files)
model_name: "my_model"
# The target word/phrase to be detected by the model. Adding multiple unique words/phrases will
# still only train a binary model detection model, but it will activate on any one of the provided words/phrases.
target_phrase:
- "hey jarvis"
# Specific phrases that you do *not* want the model to activate on, outside of those generated automatically via phoneme overlap
# This can be a good way to reduce false positives if you notice that, in practice, certain words or phrases are problematic
custom_negative_phrases: []
# The total number of positive samples to generate for training (minimum of 20,000 recommended, often 100,000+ is best)
n_samples: 10000
# The total number of positive samples to generate for validation and early stopping of model training
n_samples_val: 2000
# The batch size to use with Piper TTS when generating synthetic training data
tts_batch_size: 50
# The batch size to use when performing data augmentation on generated clips prior to training
# It's recommended that this not be too large to ensure that there is enough variety in the augmentation
augmentation_batch_size: 16
# The path to a fork of the piper-sample-generator repository for TTS (https://github.com/dscripka/piper-sample-generator)
piper_sample_generator_path: "./piper-sample-generator"
# The output directory for the generated synthetic clips, openwakeword features, and trained models
# Sub-directories will be automatically created for train and test clips for both positive and negative examples
output_dir: "./my_custom_model"
# The directories containing Room Impulse Response recordings
rir_paths:
- "./mit_rirs"
# The directories containing background audio files to mix with training data
background_paths:
- "./background_clips"
# The duplication rate for the background audio clips listed above (1 or higher). Can be useful as a way to oversample
# a particular type of background noise more relevant to a given deployment environment. Values apply in the same
# order as the background_paths list above. Only useful when multiple directories are provided above.
background_paths_duplication_rate:
- 1
# The location of pre-computed openwakeword features for false-positive validation data
# If you do not have deployment environment validation data, a good general purpose dataset with
# a reasonable mix with ~11 hours of speech, noise, and music is available here: https://huggingface.co/datasets/davidscripka/openwakeword_features
false_positive_validation_data_path: "./validation_set_features.npy"
# The number of times to apply augmentations to the generated training data
# Values greater than 1 reuse each generation that many times, producing overall unique
# clips for training due to the randomness intrinsic to the augmentation despite using
# the same original synthetic generation. Can be a useful way to increase model robustness
# without having to generate extremely large numbers of synthetic examples.
augmentation_rounds: 1
# Paths to pre-computed openwakeword features for positive and negative data. Each file must be a saved
# .npy array (see the example notebook on manually training new models for details on how to create these).
# There is no limit on the number of files but training speed will decrease as more
# data will need to be read from disk for each additional file.
# Also, there is a custom dataloader that uses memory-mapping with loading data, so the total size
# of the files is not limited by the amount of available system memory (though this will result
# in decreased training throughput depending on the speed of the underlying storage device). A fast
# NVME SSD is recommended for optimal performance.
feature_data_files:
"ACAV100M_sample": "./openwakeword_features_ACAV100M_2000_hrs_16bit.npy"
# Define the number of examples from each data file per batch. Note that the key names here
# must correspond to those define in the `feature_data_files` dictionary above (except for
# the `positive` and `adversarial_negative` keys, which are automatically defined). The sum
# of the values for each key define the total batch size for training. Initial testing indicates
# that batch sizes of 1024-4096 work well in practice.
batch_n_per_class:
"ACAV100M_sample": 1024
"adversarial_negative": 50
"positive": 50
# Define the type of size of the openwakeword model to train. Increasing the layer size
# may result in a more capable model, at the cost of decreased inference speed. The default
# value (32) seems to work well in practice for most wake words/phrases.
model_type: "dnn"
layer_size: 32
# Define training parameters. The values below are recommended defaults for most applications,
# but unique deployment environments will likely require testing to determine which values
# are the most appropriate.
# The maximum number of steps to train the model
steps: 50000
# The maximum negative weight and target false positives per hour, used to control the auto training process
# The target false positive rate may not be achieved, and adjusting the maximum negative weight may be necessary
max_negative_weight: 1500
target_false_positives_per_hour: 0.2

4
examples/detect_from_microphone.py
View file

@ -22,10 +22,10 @@ import argparse
parser=argparse.ArgumentParser()
parser.add_argument(
"--chunk_size",
help="How much audio (in samples) to predict on at once",
help="How much audio (in number of samples) to predict on at once",
type=int,
default=1280,
required=True
required=False
)
parser.add_argument(
"--model_path",

431
notebooks/automatic_model_training.ipynb Normal file
View file

@ -0,0 +1,431 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c1eab0b3",
"metadata": {
"id": "c1eab0b3"
},
"source": [
"# Introduction"
]
},
{
"cell_type": "markdown",
"id": "882058c5",
"metadata": {
"id": "882058c5"
},
"source": [
"This notebook demonstrates how to train custom openWakeWord models using pre-defined datasets and an automated process for dataset generation and training. While not guaranteed to always produce the best performing model, the methods shown in this notebook often produce baseline models with releatively strong performance.\n",
"\n",
"Manual data preparation and model training (e.g., see the [training models](training_models.ipynb) notebook) remains an option for when full control over the model development process is needed.\n",
"\n",
"At a high level, the automatic training process takes advantages of several techniques to try and produce a good model, including:\n",
"\n",
"- Early-stopping and checkpoint averaging (similar to [stochastic weight averaging](https://arxiv.org/abs/1803.05407)) to search for the best models found during training, according to the validation data\n",
"- Variable learning rates with cosine decay and multiple cycles\n",
"- Adaptive batch construction to focus on only high-loss examples when the model begins to converge, combined with gradient accumulation to ensure that batch sizes are still large enough for stable training\n",
"- Cycical weight schedules for negative examples to help the model reduce false-positive rates\n",
"\n",
"See the contents of the `train.py` file for more details."
]
},
{
"cell_type": "markdown",
"id": "e08d031b",
"metadata": {
"id": "e08d031b"
},
"source": [
"# Environment Setup"
]
},
{
"cell_type": "markdown",
"id": "aee78c37",
"metadata": {
"id": "aee78c37"
},
"source": [
"To begin, we'll need to install the requirements for training custom models. In particular, a relatively recent version of Pytorch and custom fork of the [piper-sample-generator](https://github.com/dscripka/piper-sample-generator) library for generating synthetic examples for the custom model.\n",
"\n",
"**Important Note!** Currently, automated model training is only supported on linux systems due to the requirements of the text to speech library used for synthetic sample generation (Piper). It may be possible to use Piper on Windows/Mac systems, but that has not (yet) been tested."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4b1227eb",
"metadata": {
"id": "4b1227eb"
},
"outputs": [],
"source": [
"## Environment setup\n",
"\n",
"# install piper-sample-generator (currently only supports linux systems)\n",
"!git clone https://github.com/rhasspy/piper-sample-generator\n",
"!wget -O piper-sample-generator/models/en_US-libritts_r-medium.pt 'https://github.com/rhasspy/piper-sample-generator/releases/download/v2.0.0/en_US-libritts_r-medium.pt'\n",
"!pip install piper-phonemize\n",
"\n",
"# install openwakeword (full installation to support training)\n",
"!git clone --branch auto_training https://github.com/dscripka/openwakeword\n",
"!pip install -e ./openwakeword[full]\n",
"!cd openwakeword\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d4c1056e",
"metadata": {
"ExecuteTime": {
"end_time": "2023-09-04T13:42:01.183840Z",
"start_time": "2023-09-04T13:41:59.752153Z"
},
"id": "d4c1056e"
},
"outputs": [],
"source": [
"# Imports\n",
"\n",
"import os\n",
"import numpy as np\n",
"import torch\n",
"import sys\n",
"from pathlib import Path\n",
"import uuid\n",
"import yaml\n",
"import datasets\n",
"import scipy\n",
"from tqdm import tqdm\n"
]
},
{
"cell_type": "markdown",
"id": "e9d7a05a",
"metadata": {
"id": "e9d7a05a"
},
"source": [
"# Download Data"
]
},
{
"cell_type": "markdown",
"id": "c52f75cc",
"metadata": {
"id": "c52f75cc"
},
"source": [
"When training new openWakeWord models using the automated procedure, four specific types of data are required:\n",
"\n",
"1) Synthetic examples of the target word/phrase generated with text-to-speech models\n",
"\n",
"2) Synthetic examples of adversarial words/phrases generated with text-to-speech models\n",
"\n",
"3) Room impulse reponses and noise/background audio data to augment the synthetic examples and make them more realistic\n",
"\n",
"4) Generic \"negative\" audio data that is very unlikely to contain examples of the target word/phrase in the context where the model should detect it. This data can be the original audio data, or precomputed openWakeWord features ready for model training.\n",
"\n",
"5) Validation data to use for early-stopping when training the model.\n",
"\n",
"For the purposes of this notebook, all five of these sources will either be generated manually or can be obtained from HuggingFace thanks to their excellent `datasets` library and extremely generous hosting policy. Also note that while only a portion of some datasets are downloaded, for the best possible performance it is recommended to download the entire dataset and keep a local copy for future training runs."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d25a93b1",
"metadata": {
"ExecuteTime": {
"end_time": "2023-09-04T01:07:17.746749Z",
"start_time": "2023-09-04T01:07:17.740846Z"
},
"id": "d25a93b1"
},
"outputs": [],
"source": [
"# Download room impulse responses collected by MIT\n",
"# https://mcdermottlab.mit.edu/Reverb/IR_Survey.html\n",
"\n",
"output_dir = \"./mit_rirs\"\n",
"if not os.path.exists(output_dir):\n",
" os.mkdir(output_dir)\n",
"rir_dataset = datasets.load_dataset(\"davidscripka/MIT_environmental_impulse_responses\", split=\"train\", streaming=True)\n",
"\n",
"# Save clips to 16-bit PCM wav files\n",
"for row in tqdm(rir_dataset):\n",
" name = row['audio']['path'].split('/')[-1]\n",
" scipy.io.wavfile.write(os.path.join(output_dir, name), 16000, (row['audio']['array']*32767).astype(np.int16))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c0e178b",
"metadata": {
"id": "2c0e178b"
},
"outputs": [],
"source": [
"## Download noise and background audio\n",
"\n",
"# Audioset Dataset (https://research.google.com/audioset/dataset/index.html)\n",
"# Download one part of the audioset .tar files, extract, and convert to 16khz\n",
"# For full-scale training, it's recommended to download the entire dataset from\n",
"# https://huggingface.co/datasets/agkphysics/AudioSet, and\n",
"# even potentially combine it with other background noise datasets (e.g., FSD50k, Freesound, etc.)\n",
"\n",
"if not os.path.exists(\"audioset\"):\n",
" os.mkdir(\"audioset\")\n",
"\n",
"fname = \"bal_train09.tar\"\n",
"out_dir = f\"audioset/{fname}\"\n",
"link = \"https://huggingface.co/datasets/agkphysics/AudioSet/resolve/main/\" + fname\n",
"!wget -O {out_dir} {link}\n",
"!cd audioset && tar -xvf bal_train09.tar\n",
"\n",
"output_dir = \"./audioset_16k\"\n",
"if not os.path.exists(output_dir):\n",
" os.mkdir(output_dir)\n",
"\n",
"# Convert audioset files to 16khz sample rate\n",
"audioset_dataset = datasets.Dataset.from_dict({\"audio\": [str(i) for i in Path(\"audioset/audio\").glob(\"**/*.flac\")]})\n",
"audioset_dataset = audioset_dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16000))\n",
"for row in tqdm(audioset_dataset):\n",
" name = row['audio']['path'].split('/')[-1].replace(\".flac\", \".wav\")\n",
" scipy.io.wavfile.write(os.path.join(output_dir, name), 16000, (row['audio']['array']*32767).astype(np.int16))\n",
"\n",
"# Free Music Archive dataset (https://github.com/mdeff/fma)\n",
"output_dir = \"./fma\"\n",
"if not os.path.exists(output_dir):\n",
" os.mkdir(output_dir)\n",
"fma_dataset = datasets.load_dataset(\"rudraml/fma\", name=\"small\", split=\"train\", streaming=True)\n",
"fma_dataset = iter(fma_dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16000)))\n",
"\n",
"n_hours = 1 # use only 1 hour of clips for this example notebook, recommend increasing for full-scale training\n",
"for i in tqdm(range(n_hours*3600//30)): # this works because the FMA dataset is all 30 second clips\n",
" row = next(fma_dataset)\n",
" name = row['audio']['path'].split('/')[-1].replace(\".mp3\", \".wav\")\n",
" scipy.io.wavfile.write(os.path.join(output_dir, name), 16000, (row['audio']['array']*32767).astype(np.int16))\n",
" i += 1\n",
" if i == n_hours*3600//30:\n",
" break\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d01ec467",
"metadata": {
"id": "d01ec467"
},
"outputs": [],
"source": [
"# Download pre-computed openWakeWord features for training and validation\n",
"\n",
"# training set (~2,000 hours from the ACAV100M Dataset)\n",
"# See https://huggingface.co/datasets/davidscripka/openwakeword_features for more information\n",
"!wget https://huggingface.co/datasets/davidscripka/openwakeword_features/resolve/main/openwakeword_features_ACAV100M_2000_hrs_16bit.npy\n",
"\n",
"# validation set for false positive rate estimation (~11 hours)\n",
"!wget https://huggingface.co/datasets/davidscripka/openwakeword_features/resolve/main/validation_set_features.npy"
]
},
{
"cell_type": "markdown",
"id": "cfe82647",
"metadata": {
"id": "cfe82647"
},
"source": [
"# Define Training Configuration"
]
},
{
"cell_type": "markdown",
"id": "b2e71329",
"metadata": {
"id": "b2e71329"
},
"source": [
"For automated model training openWakeWord uses a specially designed training script and a [YAML](https://yaml.org/) configuration file that defines all of the information required for training a new wake word/phrase detection model.\n",
"\n",
"It is strongly recommended that you review [the example config file](../examples/custom_model.yml), as each value is fully documented there. For the purposes of this notebook, we'll read in the YAML file to modify certain configuration parameters before saving a new YAML file for training our example model. Specifically:\n",
"\n",
"- We'll train a detection model for the phrase \"hey sebastian\"\n",
"- We'll only generate 5,000 positive and negative examples (to save on time for this example)\n",
"- We'll only generate 1,000 validation positive and negative examples for early stopping (again to save time)\n",
"- The model will only be trained for 10,000 steps (larger datasets will benefit from longer training)\n",
"- We'll reduce the target metrics to account for the small dataset size and limited training.\n",
"\n",
"On the topic of target metrics, there are *not* specific guidelines about what these metrics should be in practice, and you will need to conduct testing in your target deployment environment to establish good thresholds. However, from very limited testing the default values in the config file (accuracy >= 0.7, recall >= 0.5, false-positive rate <= 0.2 per hour) seem to produce models with reasonable performance.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fb0b6e4f",
"metadata": {
"ExecuteTime": {
"end_time": "2023-09-04T18:11:33.893397Z",
"start_time": "2023-09-04T18:11:33.878938Z"
},
"id": "fb0b6e4f"
},
"outputs": [],
"source": [
"# Load default YAML config file for training\n",
"config = yaml.load(open(\"openwakeword/examples/custom_model.yml\", 'r').read(), yaml.Loader)\n",
"config"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "482cf2d0",
"metadata": {
"ExecuteTime": {
"end_time": "2023-09-04T15:07:00.859210Z",
"start_time": "2023-09-04T15:07:00.841472Z"
},
"id": "482cf2d0"
},
"outputs": [],
"source": [
"# Modify values in the config and save a new version\n",
"\n",
"config[\"target_phrase\"] = [\"hey sebastian\"]\n",
"config[\"model_name\"] = config[\"target_phrase\"][0].replace(\" \", \"_\")\n",
"config[\"n_samples\"] = 1000\n",
"config[\"n_samples_val\"] = 1000\n",
"config[\"steps\"] = 10000\n",
"config[\"target_accuracy\"] = 0.6\n",
"config[\"target_recall\"] = 0.25\n",
"\n",
"config[\"background_paths\"] = ['./audioset_16k', './fma'] # multiple background datasets are supported\n",
"config[\"false_positive_validation_data_path\"] = \"validation_set_features.npy\"\n",
"config[\"feature_data_files\"] = {\"ACAV100M_sample\": \"openwakeword_features_ACAV100M_2000_hrs_16bit.npy\"}\n",
"\n",
"with open('my_model.yaml', 'w') as file:\n",
" documents = yaml.dump(config, file)"
]
},
{
"cell_type": "markdown",
"id": "aa6b2ab0",
"metadata": {
"id": "aa6b2ab0"
},
"source": [
"# Train the Model"
]
},
{
"cell_type": "markdown",
"id": "a51202c0",
"metadata": {
"id": "a51202c0"
},
"source": [
"With the data downloaded and training configuration set, we can now start training the model. We'll do this in parts to better illustrate the sequence, but you can also execute every step at once for a fully automated process."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f01531fa",
"metadata": {
"ExecuteTime": {
"end_time": "2023-09-04T13:50:08.803326Z",
"start_time": "2023-09-04T13:50:06.790241Z"
},
"id": "f01531fa"
},
"outputs": [],
"source": [
"# Step 1: Generate synthetic clips\n",
"# For the number of clips we are using, this should take ~10 minutes on a free Google Colab instance with a T4 GPU\n",
"# If generation fails, you can simply run this command again as it will continue generating until the\n",
"# number of files meets the targets specified in the config file\n",
"\n",
"!{sys.executable} openwakeword/openwakeword/train.py --training_config my_model.yaml --generate_clips"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "afeedae4",
"metadata": {
"ExecuteTime": {
"end_time": "2023-09-04T13:56:08.781018Z",
"start_time": "2023-09-04T13:55:40.203515Z"
},
"id": "afeedae4"
},
"outputs": [],
"source": [
"# Step 2: Augment the generated clips\n",
"\n",
"!{sys.executable} openwakeword/openwakeword/train.py --training_config my_model.yaml --augment_clips"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9ad81ea0",
"metadata": {
"ExecuteTime": {
"end_time": "2023-09-04T15:11:14.742260Z",
"start_time": "2023-09-04T15:07:03.755159Z"
},
"id": "9ad81ea0"
},
"outputs": [],
"source": [
"# Step 3: Train model\n",
"\n",
"!{sys.executable} openwakeword/openwakeword/train.py --training_config my_model.yaml --train_model"
]
},
{
"cell_type": "markdown",
"source": [
"After the model finishes training, the auto training script will automatically convert it to ONNX and tflite versions, saving them as `<model_name>.onnx/tflite` in the present working directory, where `<model_name>` is defined in the YAML training config file. Either version can be used as normal with `openwakeword`. I recommend testing them with the [`detect_from_microphone.py`](https://github.com/dscripka/openWakeWord/blob/main/examples/detect_from_microphone.py) example script to see how the model performs!"
],
"metadata": {
"id": "f9OyUW3ltOSs"
},
"id": "f9OyUW3ltOSs"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"colab": {
"provenance": []
}
},
"nbformat": 4,
"nbformat_minor": 5
}

305
openwakeword/data.py
View file

@ -15,13 +15,19 @@
# imports
from multiprocessing.pool import ThreadPool
import os
import re
import logging
from functools import partial
from pathlib import Path
import random
from tqdm import tqdm
from typing import List, Tuple
import numpy as np
import itertools
import pronouncing
import torch
import audiomentations
import torch_audiomentations
from numpy.lib.format import open_memmap
from speechbrain.dataio.dataio import read_audio
from speechbrain.processing.signal_processing import reverberate
@ -548,6 +554,181 @@ def apply_reverb(x, rir_files):
return reverbed.numpy()
# Alternate data augmentation method using audiomentations library (https://pypi.org/project/audiomentations/)
def augment_clips(
clip_paths: List[str],
total_length: int,
sr: int = 16000,
batch_size: int = 128,
augmentation_probabilities: dict = {
"SevenBandParametricEQ": 0.25,
"TanhDistortion": 0.25,
"PitchShift": 0.25,
"BandStopFilter": 0.25,
"AddColoredNoise": 0.25,
"AddBackgroundNoise": 0.75,
"Gain": 1.0,
"RIR": 0.5
},
background_clip_paths: List[str] = [],
RIR_paths: List[str] = []
):
"""
Applies audio augmentations to the specified audio clips, returning a generator that applies
the augmentations in batches to support very large quantities of input audio files.
The augmentations (and probabilities) are chosen from experience based on training openWakeWord models, as well
as for the efficiency of the augmentation. The individual probabilities of each augmentation may be adjusted
with the "augmentation_probabilities" argument.
Args:
clip_paths (List[str]) = The input audio files (as paths) to augment. Note that these should be shorter
than the "total_length" argument, else they will be truncated.
total_length (int): The total length of audio files (in samples) after augmentation. All input clips
will be left-padded with silence to reach this size, with between 0 and 200 ms
of other audio after the end of the original input clip.
sr (int): The sample size of the input audio files
batch_size (int): The number of audio files to augment at once.
augmentation_probabilities (dict): The individual probabilities of each augmentation. If all probabilities
are zero, the input audio files will simply be padded with silence. THe
default values are:
{
"SevenBandParametricEQ": 0.25,
"TanhDistortion": 0.25,
"PitchShift": 0.25,
"BandStopFilter": 0.25,
"AddColoredNoise": 0.25,
"AddBackgroundNoise": 0.75,
"Gain": 1.0,
"RIR": 0.5
}
background_clip_paths (List[str]) = The paths to background audio files to mix with the input files
RIR_paths (List[str]) = The paths to room impulse response functions (RIRs) to convolve with the input files,
producing a version of the input clip with different acoustic characteristics.
Returns:
ndarray: A batch of augmented audio clips of size (batch_size, total_length)
"""
# Define augmentations
# First pass augmentations that can't be done as a batch
augment1 = audiomentations.Compose([
audiomentations.SevenBandParametricEQ(min_gain_db=-6, max_gain_db=6, p=augmentation_probabilities["SevenBandParametricEQ"]),
audiomentations.TanhDistortion(
min_distortion=0.0001,
max_distortion=0.10,
p=augmentation_probabilities["TanhDistortion"]
),
])
# Augmentations that can be done as a batch
if background_clip_paths != []:
augment2 = torch_audiomentations.Compose([
torch_audiomentations.PitchShift(
min_transpose_semitones=-3,
max_transpose_semitones=3,
p=augmentation_probabilities["PitchShift"],
sample_rate=16000,
mode="per_batch"
),
torch_audiomentations.BandStopFilter(p=augmentation_probabilities["BandStopFilter"], mode="per_batch"),
torch_audiomentations.AddColoredNoise(
min_snr_in_db=10, max_snr_in_db=30,
min_f_decay=-1, max_f_decay=2, p=augmentation_probabilities["AddColoredNoise"],
mode="per_batch"
),
torch_audiomentations.AddBackgroundNoise(
p=augmentation_probabilities["AddBackgroundNoise"],
background_paths=background_clip_paths,
min_snr_in_db=-10,
max_snr_in_db=15,
mode="per_batch"
),
torch_audiomentations.Gain(max_gain_in_db=0, p=augmentation_probabilities["Gain"]),
])
else:
augment2 = torch_audiomentations.Compose([
torch_audiomentations.PitchShift(
min_transpose_semitones=-3,
max_transpose_semitones=3,
p=augmentation_probabilities["PitchShift"],
sample_rate=16000,
mode="per_batch"
),
torch_audiomentations.BandStopFilter(p=augmentation_probabilities["BandStopFilter"], mode="per_batch"),
torch_audiomentations.AddColoredNoise(
min_snr_in_db=10, max_snr_in_db=30,
min_f_decay=-1, max_f_decay=2, p=augmentation_probabilities["AddColoredNoise"],
mode="per_batch"
),
torch_audiomentations.Gain(max_gain_in_db=0, p=augmentation_probabilities["Gain"]),
])
# Iterate through all clips and augment them
for i in range(0, len(clip_paths), batch_size):
batch = clip_paths[i:i+batch_size]
augmented_clips = []
for clip in batch:
clip_data, clip_sr = torchaudio.load(clip)
clip_data = clip_data[0]
if clip_data.shape[0] > total_length:
clip_data = clip_data[0:total_length]
if clip_sr != sr:
raise ValueError("Error! Clip does not have the correct sample rate!")
clip_data = create_fixed_size_clip(clip_data, total_length, clip_sr)
# Do first pass augmentations
augmented_clips.append(torch.from_numpy(augment1(samples=clip_data, sample_rate=sr)))
# Do second pass augmentations
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
augmented_batch = augment2(samples=torch.vstack(augmented_clips).unsqueeze(dim=1).to(device), sample_rate=sr).squeeze(axis=1)
# Do reverberation
if augmentation_probabilities["RIR"] >= np.random.random() and RIR_paths != []:
rir_waveform, sr = torchaudio.load(random.choice(RIR_paths))
augmented_batch = reverberate(augmented_batch.cpu(), rir_waveform, rescale_amp="avg")
# yield batch of 16-bit PCM audio data
yield (augmented_batch.cpu().numpy()*32767).astype(np.int16)
def create_fixed_size_clip(x, n_samples, sr=16000, start=None, end_jitter=.200):
"""
Create a fixed-length clip of the specified size by padding an input clip with zeros
Optionally specify the start/end position of the input clip, or let it be chosen randomly.
Args:
x (ndarray): The input audio to pad to a fixed size
n_samples (int): The total number of samples for the fixed length clip
sr (int): The sample rate of the audio
start (int): The start position of the clip in the fixed length output, in samples (default: None)
end_jitter (float): The time (in seconds) from the end of the fixed length output
that the input clip should end, if `start` is None.
Returns:
ndarray: A new array of audio data of the specified length
"""
dat = np.zeros(n_samples)
end_jitter = int(np.random.uniform(0, end_jitter)*sr)
if start is None:
start = max(0, n_samples - (int(len(x))+end_jitter))
if len(x) > n_samples:
if np.random.random() >= 0.5:
dat = x[0:n_samples].numpy()
else:
dat = x[-n_samples:].numpy()
else:
dat[start:start+len(x)] = x
return dat
# Load batches of data from mmaped numpy arrays
class mmap_batch_generator:
"""
@ -645,7 +826,6 @@ class mmap_batch_generator:
# Restart at zeroth index if an array reaches the end
if self.data_counter[label] >= self.shapes[label][0]:
self.data_counter[label] = 0
# self.data[label] = np.load(self.data_files[label], mmap_mode='r')
# Get data from mmaped file
x = self.data[label][self.data_counter[label]:self.data_counter[label]+n]
@ -697,7 +877,7 @@ def trim_mmap(mmap_path):
mmap_file2 = open_memmap(output_file2, mode='w+', dtype=np.float32,
shape=(N_new, mmap_file1.shape[1], mmap_file1.shape[2]))
for i in tqdm(range(0, mmap_file1.shape[0], 1024), total=mmap_file1.shape[0]//1024):
for i in tqdm(range(0, mmap_file1.shape[0], 1024), total=mmap_file1.shape[0]//1024, desc="Trimming empty rows"):
if i + 1024 > N_new:
mmap_file2[i:N_new] = mmap_file1[i:N_new].copy()
mmap_file2.flush()
@ -710,3 +890,124 @@ def trim_mmap(mmap_path):
# Rename new mmap file to match original
os.rename(output_file2, mmap_path)
# Generate words that sound similar ("adversarial") to the input phrase using phoneme overlap
def generate_adversarial_texts(input_text: str, N: int, include_partial_phrase: float = 0, include_input_words: float = 0):
"""
Generate adversarial words and phrases based on phoneme overlap.
Currently only works for english texts.
Note that homophones are excluded, as this wouldn't actually be an adversarial example for the input text.
Args:
input_text (str): The target text for adversarial phrases
N (int): The total number of adversarial texts to return. Uses sampling,
so not all possible combinations will be included and some duplicates
may be present.
include_partial_phrase (float): The probability of returning a number of words less than the input
text (but always between 1 and the number of input words)
include_input_words (float): The probability of including individual input words in the adversarial
texts when the input text consists of multiple words. For example,
if the `input_text` was "ok google", then setting this value > 0.0
will allow for adversarial texts like "ok noodle", versus the word "ok"
never being present in the adversarial texts.
Returns:
list: A list of strings corresponding to words and phrases that are phonetically similar (but not identical)
to the input text.
"""
# Get phonemes for english vowels (CMUDICT labels)
vowel_phones = ["AA", "AE", "AH", "AO", "AW", "AX", "AXR", "AY", "EH", "ER", "EY", "IH", "IX", "IY", "OW", "OY", "UH", "UW", "UX"]
word_phones = []
input_text_phones = [pronouncing.phones_for_word(i) for i in input_text.split()]
# Download phonemizer model for OOV words, if needed
if [] in input_text_phones:
phonemizer_mdl_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "resources", "en_us_cmudict_forward.pt")
if not os.path.exists(phonemizer_mdl_path):
logging.warning("Downloading phonemizer model from DeepPhonemizer library...")
import requests
file_url = "https://public-asai-dl-models.s3.eu-central-1.amazonaws.com/DeepPhonemizer/en_us_cmudict_forward.pt"
r = requests.get(file_url, stream=True)
with open(phonemizer_mdl_path, "wb") as f:
for chunk in r.iter_content(chunk_size=2048):
if chunk:
f.write(chunk)
# Create phonemizer object
from dp.phonemizer import Phonemizer
phonemizer = Phonemizer.from_checkpoint(phonemizer_mdl_path)
for phones, word in zip(input_text_phones, input_text.split()):
if phones != []:
word_phones.extend(phones)
elif phones == []:
logging.warning(f"The word '{word}' was not found in the pronunciation dictionary! "
"Using the DeepPhonemizer library to predict the phonemes.")
phones = phonemizer(word, lang='en_us')
logging.warning(f"Phones for '{word}': {phones}")
word_phones.append(re.sub(r"[\]|\[]", "", re.sub(r"\]\[", " ", phones)))
elif isinstance(phones[0], list):
logging.warning(f"There are multiple pronunciations for the word '{word}'.")
word_phones.append(phones[0])
# add all possible lexical stresses to vowels
word_phones = [re.sub('|'.join(vowel_phones), lambda x: str(x.group(0)) + '[0|1|2]', re.sub(r'\d+', '', i)) for i in word_phones]
adversarial_phrases = []
for phones, word in zip(word_phones, input_text.split()):
query_exps = []
phones = phones.split()
adversarial_words = []
if len(phones) == 2:
query_exps.append(" ".join(phones))
else:
query_exps.extend(phoneme_replacement(phones, max_replace=max(0, len(phones)-2), replace_char="(.){1,3}"))
for query in query_exps:
matches = pronouncing.search(query)
matches_phones = [pronouncing.phones_for_word(i)[0] for i in matches]
allowed_matches = [i for i, j in zip(matches, matches_phones) if j != phones]
adversarial_words.extend([i for i in allowed_matches if word.lower() != i])
if adversarial_words != []:
adversarial_phrases.append(adversarial_words)
# Build combinations for final output
adversarial_texts = []
for i in range(N):
txts = []
for j, k in zip(adversarial_phrases, input_text.split()):
if np.random.random() > (1 - include_input_words):
txts.append(k)
else:
txts.append(np.random.choice(j))
if include_partial_phrase is not None and len(input_text.split()) > 1 and np.random.random() <= include_partial_phrase:
n_words = np.random.randint(1, len(input_text.split())+1)
adversarial_texts.append(" ".join(np.random.choice(txts, size=n_words, replace=False)))
else:
adversarial_texts.append(" ".join(txts))
# Remove any exact matches to input phrase
adversarial_texts = [i for i in adversarial_texts if i != input_text]
return adversarial_texts
def phoneme_replacement(input_chars, max_replace, replace_char='"(.){1,3}"'):
results = []
chars = list(input_chars)
# iterate over the number of characters to replace (1 to max_replace)
for r in range(1, max_replace+1):
# get all combinations for a fixed r
comb = itertools.combinations(range(len(chars)), r)
for indices in comb:
chars_copy = chars.copy()
for i in indices:
chars_copy[i] = replace_char
results.append(' '.join(chars_copy))
return results

753
openwakeword/train.py Executable file
View file

@ -0,0 +1,753 @@
import torch
from torch import optim, nn
import torchinfo
import torchmetrics
import copy
import os
import sys
import tempfile
import uuid
import numpy as np
import scipy
import collections
import argparse
import logging
from tqdm import tqdm
import yaml
from pathlib import Path
import openwakeword
from openwakeword.data import generate_adversarial_texts, augment_clips, mmap_batch_generator
from openwakeword.utils import compute_features_from_generator
# Base model class for an openwakeword model
class Model(nn.Module):
def __init__(self, n_classes=1, input_shape=(16, 96), model_type="dnn",
layer_dim=128, seconds_per_example=None):
super().__init__()
# Store inputs as attributes
self.n_classes = n_classes
self.input_shape = input_shape
self.seconds_per_example = seconds_per_example
self.device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
self.best_models = []
self.best_model_scores = []
self.best_val_fp = 1000
self.best_val_accuracy = 0
self.best_val_recall = 0
self.best_train_recall = 0
# Define model (currently on fully-connected network supported)
if model_type == "dnn":
self.model = nn.Sequential(
nn.Flatten(),
nn.Linear(input_shape[0]*input_shape[1], layer_dim),
nn.LayerNorm(layer_dim),
nn.ReLU(),
nn.Linear(layer_dim, layer_dim),
nn.LayerNorm(layer_dim),
nn.ReLU(),
nn.Linear(layer_dim, n_classes),
nn.Sigmoid() if n_classes == 1 else nn.ReLU(),
)
elif model_type == "rnn":
class Net(nn.Module):
def __init__(self, input_shape, n_classes=1):
super().__init__()
self.layer1 = nn.LSTM(input_shape[-1], 64, num_layers=2, bidirectional=True,
batch_first=True, dropout=0.0)
self.layer2 = nn.Linear(64*2, n_classes)
self.layer3 = nn.Sigmoid() if n_classes == 1 else nn.ReLU()
def forward(self, x):
out, h = self.layer1(x)
return self.layer3(self.layer2(out[:, -1]))
self.model = Net(input_shape, n_classes)
# Define metrics
if n_classes == 1:
self.fp = lambda pred, y: (y-pred <= -0.5).sum()
self.recall = torchmetrics.Recall(task='binary')
self.accuracy = torchmetrics.Accuracy(task='binary')
else:
def multiclass_fp(p, y, threshold=0.5):
probs = torch.nn.functional.softmax(p, dim=1)
neg_ndcs = y == 0
fp = (probs[neg_ndcs].argmax(axis=1) != 0 & (probs[neg_ndcs].max(axis=1)[0] > threshold)).sum()
return fp
def positive_class_recall(p, y, negative_class_label=0, threshold=0.5):
probs = torch.nn.functional.softmax(p, dim=1)
pos_ndcs = y != 0
rcll = (probs[pos_ndcs].argmax(axis=1) > 0
& (probs[pos_ndcs].max(axis=1)[0] >= threshold)).sum()/pos_ndcs.sum()
return rcll
def positive_class_accuracy(p, y, negative_class_label=0):
probs = torch.nn.functional.softmax(p, dim=1)
pos_preds = probs.argmax(axis=1) != negative_class_label
acc = (probs[pos_preds].argmax(axis=1) == y[pos_preds]).sum()/pos_preds.sum()
return acc
self.fp = multiclass_fp
self.acc = positive_class_accuracy
self.recall = positive_class_recall
self.n_fp = 0
self.val_fp = 0
# Define logging dict (in-memory)
self.history = collections.defaultdict(list)
# Define optimizer and loss
self.loss = torch.nn.functional.binary_cross_entropy if n_classes == 1 else nn.functional.cross_entropy
self.optimizer = optim.Adam(self.model.parameters(), lr=0.0001)
def save_model(self, output_path):
"""
Saves the weights of a trained Pytorch model
"""
if self.n_classes == 1:
torch.save(self.model, output_path)
def export_to_onnx(self, output_path, class_mapping=""):
obj = self
# Make simple model for export based on model structure
if self.n_classes == 1:
# Save ONNX model
torch.onnx.export(self.model.to("cpu"), torch.rand(self.input_shape)[None, ], output_path,
output_names=[class_mapping])
elif self.n_classes >= 1:
class M(nn.Module):
def __init__(self):
super().__init__()
# Define model
self.model = obj.model.to("cpu")
def forward(self, x):
return torch.nn.functional.softmax(self.model(x), dim=1)
# Save ONNX model
torch.onnx.export(M(), torch.rand(self.input_shape)[None, ], output_path,
output_names=[class_mapping])
def lr_warmup_cosine_decay(self,
global_step,
warmup_steps=0,
hold=0,
total_steps=0,
start_lr=0.0,
target_lr=1e-3
):
# Cosine decay
learning_rate = 0.5 * target_lr * (1 + np.cos(np.pi * (global_step - warmup_steps - hold)
/ float(total_steps - warmup_steps - hold)))
# Target LR * progress of warmup (=1 at the final warmup step)
warmup_lr = target_lr * (global_step / warmup_steps)
# Choose between `warmup_lr`, `target_lr` and `learning_rate` based on whether
# `global_step < warmup_steps` and we're still holding.
# i.e. warm up if we're still warming up and use cosine decayed lr otherwise
if hold > 0:
learning_rate = np.where(global_step > warmup_steps + hold,
learning_rate, target_lr)
learning_rate = np.where(global_step < warmup_steps, warmup_lr, learning_rate)
return learning_rate
def forward(self, x):
return self.model(x)
def summary(self):
return torchinfo.summary(self.model, input_size=(1,) + self.input_shape)
def average_models(self, models=None):
"""Averages the weights of the provided models together to make a new model"""
if models is None:
models = self.best_models
# Clone a model from the list as the base for the averaged model
averaged_model = copy.deepcopy(models[0])
averaged_model_dict = averaged_model.state_dict()
# Initialize a running total of the weights
for key in averaged_model_dict:
averaged_model_dict[key] *= 0 # set to 0
for model in models:
model_dict = model.state_dict()
for key, value in model_dict.items():
averaged_model_dict[key] += value
for key in averaged_model_dict:
averaged_model_dict[key] /= len(models)
# Load the averaged weights into the model
averaged_model.load_state_dict(averaged_model_dict)
return averaged_model
def auto_train(self, X_train, X_val, false_positive_val_data, steps=50000, max_negative_weight=1000,
target_fp_per_hour=0.2):
"""A sequence of training steps that produce relatively strong models
automatically, based on validation data and performance targets provided.
After training merges the best checkpoints and returns a single model.
"""
# Get false positive validation data duration
val_set_hrs = 11.3
# Sequence 1
logging.info("#"*50 + "\nStarting training sequence 1...\n" + "#"*50)
lr = 0.0001
weights = np.linspace(1, max_negative_weight, int(steps)).tolist()
val_steps = np.linspace(steps-int(steps*0.25), steps, 20).astype(np.int64)
self.train_model(
X=X_train,
X_val=X_val,
false_positive_val_data=false_positive_val_data,
max_steps=steps,
negative_weight_schedule=weights,
val_steps=val_steps, warmup_steps=steps//5,
hold_steps=steps//3, lr=lr, val_set_hrs=val_set_hrs)
# Sequence 2
logging.info("#"*50 + "\nStarting training sequence 2...\n" + "#"*50)
lr = lr/10
steps = steps/10
# Adjust weights as needed based on false positive per hour performance from first sequence
if self.best_val_fp > target_fp_per_hour:
max_negative_weight = max_negative_weight*2
logging.info("Increasing weight on negative examples to reduce false positives...")
weights = np.linspace(1, max_negative_weight, int(steps)).tolist()
val_steps = np.linspace(1, steps, 20).astype(np.int16)
self.train_model(
X=X_train,
X_val=X_val,
false_positive_val_data=false_positive_val_data,
max_steps=steps,
negative_weight_schedule=weights,
val_steps=val_steps, warmup_steps=steps//5,
hold_steps=steps//3, lr=lr, val_set_hrs=val_set_hrs)
# Sequence 3
logging.info("#"*50 + "\nStarting training sequence 3...\n" + "#"*50)
lr = lr/10
# Adjust weights as needed based on false positive per hour performance from second sequence
if self.best_val_fp > target_fp_per_hour:
max_negative_weight = max_negative_weight*2
logging.info("Increasing weight on negative examples to reduce false positives...")
weights = np.linspace(1, max_negative_weight, int(steps)).tolist()
val_steps = np.linspace(1, steps, 20).astype(np.int16)
self.train_model(
X=X_train,
X_val=X_val,
false_positive_val_data=false_positive_val_data,
max_steps=steps,
negative_weight_schedule=weights,
val_steps=val_steps, warmup_steps=steps//5,
hold_steps=steps//3, lr=lr, val_set_hrs=val_set_hrs)
# Merge best models
logging.info("Merging checkpoints above the 90th percentile into single model...")
accuracy_percentile = np.percentile(self.history["val_accuracy"], 90)
recall_percentile = np.percentile(self.history["val_recall"], 90)
fp_percentile = np.percentile(self.history["val_fp_per_hr"], 10)
# Get models above the 90th percentile
models = []
for model, score in zip(self.best_models, self.best_model_scores):
if score["val_accuracy"] >= accuracy_percentile and \
score["val_recall"] >= recall_percentile and \
score["val_fp_per_hr"] <= fp_percentile:
models.append(model)
if len(models) > 0:
combined_model = self.average_models(models=models)
else:
combined_model = self.model
# Report validation metrics for combined model
with torch.no_grad():
for batch in X_val:
x, y = batch[0].to(self.device), batch[1].to(self.device)
val_ps = combined_model(x)
combined_model_recall = self.recall(val_ps, y[..., None]).detach().cpu().numpy()
combined_model_accuracy = self.accuracy(val_ps, y[..., None].to(torch.int64)).detach().cpu().numpy()
combined_model_fp = 0
for batch in false_positive_val_data:
x_val, y_val = batch[0].to(self.device), batch[1].to(self.device)
val_ps = combined_model(x_val)
combined_model_fp += self.fp(val_ps, y_val[..., None])
combined_model_fp_per_hr = (combined_model_fp/val_set_hrs).detach().cpu().numpy()
logging.info(f"\n################\nFinal Model Accuracy: {combined_model_accuracy}"
f"\nFinal Model Recall: {combined_model_recall}\nFinal Model False Positives per Hour: {combined_model_fp_per_hr}"
"\n################\n")
return combined_model
def export_model(self, model, model_name, output_dir):
"""Saves the trained openwakeword model to both onnx and tflite formats"""
if self.n_classes != 1:
raise ValueError("Exporting models to both onnx and tflite with more than one class is currently not supported! "
"Use the `export_to_onnx` function instead.")
# Save ONNX model
logging.info(f"####\nSaving ONNX mode as '{os.path.join(output_dir, model_name + '.onnx')}'")
model_to_save = copy.deepcopy(model)
torch.onnx.export(model_to_save.to("cpu"), torch.rand(self.input_shape)[None, ], os.path.join(output_dir, model_name + ".onnx"))
return None
def train_model(self, X, max_steps, warmup_steps, hold_steps, X_val=None,
false_positive_val_data=None,
negative_weight_schedule=[1],
val_steps=[250], lr=0.0001, val_set_hrs=1):
# Move models and main class to target device
self.to(self.device)
self.model.to(self.device)
# Train model
accumulation_steps = 1
accumulated_samples = 0
for step_ndx, data in tqdm(enumerate(X, 0), total=max_steps, desc="Training"):
# get the inputs; data is a list of [inputs, labels]
x, y = data[0].to(self.device), data[1].to(self.device)
y_ = y[..., None].to(torch.float32)
# Update learning rates
for g in self.optimizer.param_groups:
g['lr'] = self.lr_warmup_cosine_decay(step_ndx, warmup_steps=warmup_steps, hold=hold_steps,
total_steps=max_steps, target_lr=lr)
# zero the parameter gradients
self.optimizer.zero_grad()
# Get predictions for batch
predictions = self.model(x)
# Construct batch with only samples that have high loss
neg_high_loss = predictions[(y == 0) & (predictions.squeeze() >= 0.001)] # thresholds were chosen arbitrarily but work well
pos_high_loss = predictions[(y == 1) & (predictions.squeeze() < 0.999)]
y = torch.cat((y[(y == 0) & (predictions.squeeze() >= 0.001)], y[(y == 1) & (predictions.squeeze() < 0.999)]))
y_ = y[..., None].to(torch.float32)
predictions = torch.cat((neg_high_loss, pos_high_loss))
# Set weights for batch
if len(negative_weight_schedule) == 1:
w = torch.ones(y.shape[0])*negative_weight_schedule[0]
pos_ndcs = y == 1
w[pos_ndcs] = 1
w = w[..., None]
else:
if self.n_classes == 1:
w = torch.ones(y.shape[0])*negative_weight_schedule[step_ndx]
pos_ndcs = y == 1
w[pos_ndcs] = 1
w = w[..., None]
# Do backpropagation, with gradient accumulation if the batch-size after selecting high loss examples is too small
loss = self.loss(predictions, y_ if self.n_classes == 1 else y, w.to(self.device))
loss = loss/accumulation_steps
accumulated_samples += predictions.shape[0]
if accumulated_samples < 128:
accumulation_steps += 1
else:
loss.backward()
self.optimizer.step()
accumulation_steps = 1
accumulated_samples = 0
# Compute training metrics and log them
fp = self.fp(predictions, y_ if self.n_classes == 1 else y)
self.n_fp += fp
self.history["loss"].append(loss.detach().cpu().numpy())
self.history["recall"].append(self.recall(predictions, y_).detach().cpu().numpy())
if self.n_classes != 1:
self.history["accuracy"].append(self.acc(predictions, y).detach().cpu().numpy())
# Run validation and log validation metrics
if step_ndx in val_steps and step_ndx > 1 and false_positive_val_data is not None:
# Get false positives per hour with false positive data
val_fp = 0
for val_step_ndx, data in enumerate(false_positive_val_data):
with torch.no_grad():
x_val, y_val = data[0].to(self.device), data[1].to(self.device)
val_predictions = self.model(x_val)
val_fp += self.fp(val_predictions, y_val[..., None])
val_fp_per_hr = (val_fp/val_set_hrs).detach().cpu().numpy()
self.history["val_fp_per_hr"].append(val_fp_per_hr)
if step_ndx in val_steps and step_ndx > 1 and X_val is not None:
# Get accuracy for balanced test examples of positive and negative clips
for val_step_ndx, data in enumerate(X_val):
with torch.no_grad():
x_val, y_val = data[0].to(self.device), data[1].to(self.device)
val_predictions = self.model(x_val)
val_recall = self.recall(val_predictions, y_val[..., None]).detach().cpu().numpy()
val_acc = self.accuracy(val_predictions, y_val[..., None].to(torch.int64))
self.history["val_accuracy"].append(val_acc.detach().cpu().numpy())
self.history["val_recall"].append(val_recall)
# Save models with a validation score above/below the 90th percentile
# of the validation scores up to that point
if val_fp_per_hr <= np.percentile(self.history["val_fp_per_hr"], 10) and \
self.history["val_accuracy"][-1] >= np.percentile(self.history["val_accuracy"], 90) and \
self.history["val_recall"][-1] >= np.percentile(self.history["val_recall"], 90):
# logging.info("Saving checkpoint with metrics >= to targets!")
self.best_models.append(copy.deepcopy(self.model))
self.best_model_scores.append({"val_fp_per_hr": val_fp_per_hr, "val_accuracy": self.history["val_accuracy"][-1],
"val_recall": self.history["val_recall"][-1]})
self.best_val_fp = val_fp_per_hr
self.best_val_recall = self.history["val_recall"][-1]
self.best_val_accuracy = self.history["val_accuracy"][-1]
if step_ndx == max_steps-1:
break
# Separate function to convert onnx models to tflite format
def convert_onnx_to_tflite(onnx_model_path, output_path):
"""Converts an ONNX version of an openwakeword model to the Tensorflow tflite format."""
# imports
import onnx
from onnx_tf.backend import prepare
import tensorflow as tf
# Convert to tflite from onnx model
onnx_model = onnx.load(onnx_model_path)
tf_rep = prepare(onnx_model, device="CPU")
with tempfile.TemporaryDirectory() as tmp_dir:
tf_rep.export_graph(os.path.join(tmp_dir, "tf_model"))
converter = tf.lite.TFLiteConverter.from_saved_model(os.path.join(tmp_dir, "tf_model"))
tflite_model = converter.convert()
logging.info(f"####\nSaving tflite mode to '{output_path}'")
with open(output_path, 'wb') as f:
f.write(tflite_model)
return None
if __name__ == '__main__':
# Get training config file
parser = argparse.ArgumentParser()
parser.add_argument(
"--training_config",
help="The path to the training config file (required)",
type=str,
required=True
)
parser.add_argument(
"--generate_clips",
help="Execute the synthetic data generation process",
action="store_true",
default="False",
required=False
)
parser.add_argument(
"--augment_clips",
help="Execute the synthetic data augmentation process",
action="store_true",
default="False",
required=False
)
parser.add_argument(
"--overwrite",
help="Overwrite existing openwakeword features when the --augment_clips flag is used",
action="store_true",
default="False",
required=False
)
parser.add_argument(
"--train_model",
help="Execute the model training process",
action="store_true",
default="False",
required=False
)
args = parser.parse_args()
config = yaml.load(open(args.training_config, 'r').read(), yaml.Loader)
# imports Piper for synthetic sample generation
sys.path.insert(0, os.path.abspath(config["piper_sample_generator_path"]))
from generate_samples import generate_samples
# Define output locations
config["output_dir"] = os.path.abspath(config["output_dir"])
if not os.path.exists(config["output_dir"]):
os.mkdir(config["output_dir"])
if not os.path.exists(os.path.join(config["output_dir"], config["model_name"])):
os.mkdir(os.path.join(config["output_dir"], config["model_name"]))
positive_train_output_dir = os.path.join(config["output_dir"], config["model_name"], "positive_train")
positive_test_output_dir = os.path.join(config["output_dir"], config["model_name"], "positive_test")
negative_train_output_dir = os.path.join(config["output_dir"], config["model_name"], "negative_train")
negative_test_output_dir = os.path.join(config["output_dir"], config["model_name"], "negative_test")
feature_save_dir = os.path.join(config["output_dir"], config["model_name"])
# Get paths for impulse response and background audio files
rir_paths = [i.path for j in config["rir_paths"] for i in os.scandir(j)]
background_paths = []
if len(config["background_paths_duplication_rate"]) != len(config["background_paths"]):
config["background_paths_duplication_rate"] = [1]*len(config["background_paths"])
for background_path, duplication_rate in zip(config["background_paths"], config["background_paths_duplication_rate"]):
background_paths.extend([i.path for i in os.scandir(background_path)]*duplication_rate)
if args.generate_clips is True:
# Generate positive clips for training
logging.info("#"*50 + "\nGenerating positive clips for training\n" + "#"*50)
if not os.path.exists(positive_train_output_dir):
os.mkdir(positive_train_output_dir)
n_current_samples = len(os.listdir(positive_train_output_dir))
if n_current_samples <= 0.95*config["n_samples"]:
generate_samples(
text=config["target_phrase"], max_samples=config["n_samples"]-n_current_samples,
batch_size=config["tts_batch_size"],
noise_scales=[0.98], noise_scale_ws=[0.98], length_scales=[0.75, 1.0, 1.25],
output_dir=positive_train_output_dir, auto_reduce_batch_size=True,
file_names=[uuid.uuid4().hex + ".wav" for i in range(config["n_samples"])]
)
torch.cuda.empty_cache()
else:
logging.warning(f"Skipping generation of positive clips for training, as ~{config['n_samples']} already exist")
# Generate positive clips for testing
logging.info("#"*50 + "\nGenerating positive clips for testing\n" + "#"*50)
if not os.path.exists(positive_test_output_dir):
os.mkdir(positive_test_output_dir)
n_current_samples = len(os.listdir(positive_test_output_dir))
if n_current_samples <= 0.95*config["n_samples_val"]:
generate_samples(text=config["target_phrase"], max_samples=config["n_samples_val"]-n_current_samples,
batch_size=config["tts_batch_size"],
noise_scales=[1.0], noise_scale_ws=[1.0], length_scales=[0.75, 1.0, 1.25],
output_dir=positive_test_output_dir, auto_reduce_batch_size=True)
torch.cuda.empty_cache()
else:
logging.warning(f"Skipping generation of positive clips testing, as ~{config['n_samples_val']} already exist")
# Generate adversarial negative clips for training
logging.info("#"*50 + "\nGenerating negative clips for training\n" + "#"*50)
if not os.path.exists(negative_train_output_dir):
os.mkdir(negative_train_output_dir)
n_current_samples = len(os.listdir(negative_train_output_dir))
if n_current_samples <= 0.95*config["n_samples"]:
adversarial_texts = config["custom_negative_phrases"]
for target_phrase in config["target_phrase"]:
adversarial_texts.extend(generate_adversarial_texts(
input_text=target_phrase,
N=config["n_samples"],
include_partial_phrase=1.0,
include_input_words=0.2))
generate_samples(text=adversarial_texts, max_samples=config["n_samples"]-n_current_samples,
batch_size=config["tts_batch_size"]//7,
noise_scales=[0.98], noise_scale_ws=[0.98], length_scales=[0.75, 1.0, 1.25],
output_dir=negative_train_output_dir, auto_reduce_batch_size=True,
file_names=[uuid.uuid4().hex + ".wav" for i in range(config["n_samples"])]
)
torch.cuda.empty_cache()
else:
logging.warning(f"Skipping generation of negative clips for training, as ~{config['n_samples']} already exist")
# Generate adversarial negative clips for testing
logging.info("#"*50 + "\nGenerating negative clips for testing\n" + "#"*50)
if not os.path.exists(negative_test_output_dir):
os.mkdir(negative_test_output_dir)
n_current_samples = len(os.listdir(negative_test_output_dir))
if n_current_samples <= 0.95*config["n_samples_val"]:
adversarial_texts = config["custom_negative_phrases"]
for target_phrase in config["target_phrase"]:
adversarial_texts.extend(generate_adversarial_texts(
input_text=target_phrase,
N=config["n_samples_val"],
include_partial_phrase=1.0,
include_input_words=0.2))
generate_samples(text=adversarial_texts, max_samples=config["n_samples_val"]-n_current_samples,
batch_size=config["tts_batch_size"]//7,
noise_scales=[1.0], noise_scale_ws=[1.0], length_scales=[0.75, 1.0, 1.25],
output_dir=negative_test_output_dir, auto_reduce_batch_size=True)
torch.cuda.empty_cache()
else:
logging.warning(f"Skipping generation of negative clips for testing, as ~{config['n_samples_val']} already exist")
# Set the total length of the training clips based on the ~median generated clip duration, rounding to the nearest 1000 samples
# and setting to 32000 when the median + 750 ms is close to that, as it's a good default value
n = 50 # sample size
positive_clips = [str(i) for i in Path(positive_test_output_dir).glob("*.wav")]
duration_in_samples = []
for i in range(n):
sr, dat = scipy.io.wavfile.read(positive_clips[np.random.randint(0, len(positive_clips))])
duration_in_samples.append(len(dat))
config["total_length"] = int(round(np.median(duration_in_samples)/1000)*1000) + 12000 # add 750 ms to clip duration as buffer
if config["total_length"] < 32000:
config["total_length"] = 32000 # set a minimum of 32000 samples (2 seconds)
elif abs(config["total_length"] - 32000) <= 4000:
config["total_length"] = 32000
# Do Data Augmentation
if args.augment_clips is True:
if not os.path.exists(os.path.join(feature_save_dir, "positive_features_train.npy")) or args.overwrite is True:
positive_clips_train = [str(i) for i in Path(positive_train_output_dir).glob("*.wav")]*config["augmentation_rounds"]
positive_clips_train_generator = augment_clips(positive_clips_train, total_length=config["total_length"],
batch_size=config["augmentation_batch_size"],
background_clip_paths=background_paths,
RIR_paths=rir_paths)
positive_clips_test = [str(i) for i in Path(positive_test_output_dir).glob("*.wav")]*config["augmentation_rounds"]
positive_clips_test_generator = augment_clips(positive_clips_test, total_length=config["total_length"],
batch_size=config["augmentation_batch_size"],
background_clip_paths=background_paths,
RIR_paths=rir_paths)
negative_clips_train = [str(i) for i in Path(negative_train_output_dir).glob("*.wav")]*config["augmentation_rounds"]
negative_clips_train_generator = augment_clips(negative_clips_train, total_length=config["total_length"],
batch_size=config["augmentation_batch_size"],
background_clip_paths=background_paths,
RIR_paths=rir_paths)
negative_clips_test = [str(i) for i in Path(negative_test_output_dir).glob("*.wav")]*config["augmentation_rounds"]
negative_clips_test_generator = augment_clips(negative_clips_test, total_length=config["total_length"],
batch_size=config["augmentation_batch_size"],
background_clip_paths=background_paths,
RIR_paths=rir_paths)
# Compute features and save to disk via memmapped arrays
logging.info("#"*50 + "\nComputing openwakeword features for generated samples\n" + "#"*50)
n_cpus = os.cpu_count()
if n_cpus is None:
n_cpus = 1
else:
n_cpus = n_cpus//2
compute_features_from_generator(positive_clips_train_generator, n_total=len(os.listdir(positive_train_output_dir)),
clip_duration=config["total_length"],
output_file=os.path.join(feature_save_dir, "positive_features_train.npy"),
device="gpu" if torch.cuda.is_available() else "cpu",
ncpu=n_cpus if not torch.cuda.is_available() else 1)
compute_features_from_generator(negative_clips_train_generator, n_total=len(os.listdir(negative_train_output_dir)),
clip_duration=config["total_length"],
output_file=os.path.join(feature_save_dir, "negative_features_train.npy"),
device="gpu" if torch.cuda.is_available() else "cpu",
ncpu=n_cpus if not torch.cuda.is_available() else 1)
compute_features_from_generator(positive_clips_test_generator, n_total=len(os.listdir(positive_test_output_dir)),
clip_duration=config["total_length"],
output_file=os.path.join(feature_save_dir, "positive_features_test.npy"),
device="gpu" if torch.cuda.is_available() else "cpu",
ncpu=n_cpus if not torch.cuda.is_available() else 1)
compute_features_from_generator(negative_clips_test_generator, n_total=len(os.listdir(negative_test_output_dir)),
clip_duration=config["total_length"],
output_file=os.path.join(feature_save_dir, "negative_features_test.npy"),
device="gpu" if torch.cuda.is_available() else "cpu",
ncpu=n_cpus if not torch.cuda.is_available() else 1)
else:
logging.warning("Openwakeword features already exist, skipping data augmentation and feature generation")
# Create openwakeword model
if args.train_model is True:
F = openwakeword.utils.AudioFeatures(device='cpu')
input_shape = F.get_embedding_shape(config["total_length"]//16000) # training data is always 16 khz
oww = Model(n_classes=1, input_shape=input_shape, model_type=config["model_type"],
layer_dim=config["layer_size"], seconds_per_example=1280*input_shape[0]/16000)
# Create data transform function for batch generation to handle differ clip lengths (todo: write tests for this)
def f(x, n=16):
"""Simple transformation function to ensure negative data is the appropriate shape for the model size"""
if n > x.shape[1] or n < x.shape[1]:
x = np.vstack(x)
new_batch = np.array([x[i:i+n, :] for i in range(0, x.shape[0]-n, n)])
else:
return x
return new_batch
# Create label transforms as needed for model (currently only supports binary classification models)
data_transforms = {key: f for key in config["feature_data_files"].keys()}
label_transforms = {}
for key in ["positive"] + list(config["feature_data_files"].keys()) + ["adversarial_negative"]:
if key == "positive":
label_transforms[key] = lambda x: [1 for i in x]
else:
label_transforms[key] = lambda x: [0 for i in x]
# Add generated positive and adversarial negative clips to the feature data files dictionary
config["feature_data_files"]['positive'] = os.path.join(feature_save_dir, "positive_features_train.npy")
config["feature_data_files"]['adversarial_negative'] = os.path.join(feature_save_dir, "negative_features_train.npy")
# Make PyTorch data loaders for training and validation data
batch_generator = mmap_batch_generator(
config["feature_data_files"],
n_per_class=config["batch_n_per_class"],
data_transform_funcs=data_transforms,
label_transform_funcs=label_transforms
)
class IterDataset(torch.utils.data.IterableDataset):
def __init__(self, generator):
self.generator = generator
def __iter__(self):
return self.generator
n_cpus = os.cpu_count()
if n_cpus is None:
n_cpus = 1
else:
n_cpus = n_cpus//2
X_train = torch.utils.data.DataLoader(IterDataset(batch_generator),
batch_size=None, num_workers=n_cpus, prefetch_factor=16)
X_val_fp = np.load(config["false_positive_validation_data_path"])
X_val_fp = np.array([X_val_fp[i:i+input_shape[0]] for i in range(0, X_val_fp.shape[0]-input_shape[0], 1)]) # reshape to match model
X_val_fp_labels = np.zeros(X_val_fp.shape[0]).astype(np.float32)
X_val_fp = torch.utils.data.DataLoader(
torch.utils.data.TensorDataset(torch.from_numpy(X_val_fp), torch.from_numpy(X_val_fp_labels)),
batch_size=len(X_val_fp_labels)
)
X_val_pos = np.load(os.path.join(feature_save_dir, "positive_features_test.npy"))
X_val_neg = np.load(os.path.join(feature_save_dir, "negative_features_test.npy"))
labels = np.hstack((np.ones(X_val_pos.shape[0]), np.zeros(X_val_neg.shape[0]))).astype(np.float32)
X_val = torch.utils.data.DataLoader(
torch.utils.data.TensorDataset(
torch.from_numpy(np.vstack((X_val_pos, X_val_neg))),
torch.from_numpy(labels)
),
batch_size=len(labels)
)
# Run auto training
best_model = oww.auto_train(
X_train=X_train,
X_val=X_val,
false_positive_val_data=X_val_fp,
steps=config["steps"],
max_negative_weight=config["max_negative_weight"],
target_fp_per_hour=config["target_false_positives_per_hour"],
)
# Export the trained model to onnx
oww.export_model(model=best_model, model_name=config["model_name"], output_dir=config["output_dir"])
# Convert the model from onnx to tflite format
convert_onnx_to_tflite(os.path.join(config["output_dir"], config["model_name"] + ".onnx"),
os.path.join(config["output_dir"], config["model_name"] + ".tflite"))

67
openwakeword/utils.py
View file

@ -21,10 +21,11 @@ from multiprocessing.pool import ThreadPool
from multiprocessing import Process, Queue
import time
import logging
from tqdm import tqdm
import openwakeword
from numpy.lib.format import open_memmap
from typing import Union, List, Callable, Deque
import requests
from tqdm import tqdm
# Base class for computing audio features using Google's speech_embedding
@ -528,9 +529,71 @@ def bulk_predict(
return {list(i.keys())[0]: list(i.values())[0] for i in results}
def compute_features_from_generator(generator, n_total, clip_duration, output_file, device="cpu", ncpu=1):
"""
Computes audio features from a generator that produces Numpy arrays of shape (batch_size, samples)
containing 16-bit PCM audio data.
Args:
generator (Generator): The generator that process the arrays of audio data
n_total (int): The total number of rows (audio clips) that the generator will produce.
Ideally this is precise, but it can be approximate as well as the output
.npy file will be automatically trimmed to remove empty values.
clip_duration (float): The duration (in samples) of the audio produced by the generator
output_file (str): The output file (.npy) containing the audio features. Note that this file
will be written to using memmap arrays, so it can be substantially larger
than the available system memory.
device (str): The device ("cpu" or "gpu") to use for computing features.
ncpu (int): The number of cores to use when process the audio features (if computing on CPU)
Returns:
None
"""
# Function specific imports
from openwakeword.data import trim_mmap
# Create audio features object
F = AudioFeatures(device=device)
# Determine the output shape and create output file
n_feature_cols = F.get_embedding_shape(clip_duration/16000)
output_shape = (n_total, n_feature_cols[0], n_feature_cols[1])
fp = open_memmap(output_file, mode='w+', dtype=np.float32, shape=output_shape)
# Get batch size by pulling one value from the generator and store features
row_counter = 0
audio_data = next(generator)
batch_size = audio_data.shape[0]
if batch_size > n_total:
raise ValueError(f"The value of 'n_total' ({n_total}) is less than the batch size ({batch_size})."
" Please increase 'n_total' to be >= batch size.")
features = F.embed_clips(audio_data, batch_size=batch_size)
fp[row_counter:row_counter+features.shape[0], :, :] = features
row_counter += features.shape[0]
fp.flush()
# Compute features and add data to output file
for audio_data in tqdm(generator, total=n_total//batch_size, desc="Computing features"):
if row_counter >= n_total:
break
features = F.embed_clips(audio_data, batch_size=batch_size, ncpu=ncpu)
if row_counter + features.shape[0] > n_total:
features = features[0:n_total-row_counter]
fp[row_counter:row_counter+features.shape[0], :, :] = features
row_counter += features.shape[0]
fp.flush()
# Trip empty rows from the mmapped array
trim_mmap(output_file)
# Function to download files from a URL with a progress bar
def download_file(url, target_directory, file_size=None):
"""A simpel function to download a file from a URL with a progress bar using only the requests library"""
"""A simple function to download a file from a URL with a progress bar using only the requests library"""
local_filename = url.split('/')[-1]
with requests.get(url, stream=True) as r:

8
openwakeword/vad.py
View file

@ -63,18 +63,20 @@ class VAD():
"resources",
"models",
"silero_vad.onnx"
)
),
n_threads: int = 1
):
"""Initialize the VAD model object.
Args:
model_path (str): The path to the Silero VAD ONNX model.
n_threads (int): The number of threads to use for the VAD model.
"""
# Initialize the ONNX model
sessionOptions = ort.SessionOptions()
sessionOptions.inter_op_num_threads = 1
sessionOptions.intra_op_num_threads = 1
sessionOptions.inter_op_num_threads = n_threads
sessionOptions.intra_op_num_threads = n_threads
self.model = ort.InferenceSession(model_path, sess_options=sessionOptions,
providers=["CPUExecutionProvider"])

23
setup.py
View file

@ -42,19 +42,36 @@ setuptools.setup(
'pytest-flake8>=1.1.1,<2',
'flake8>=4.0,<4.1',
'pytest-mypy>=0.10.0,<1',
'types-requests',
'types-PyYAML',
'mock>=5.1,<6',
'types-mock>=5.1,<6',
'types-requests>=2.0,<3'
],
'full': [
'mutagen>=1.46.0,<2',
'speechbrain>=0.5.13,<1',
'torch>=1.13.1,<2',
'torchaudio>=0.13.1,<1',
'torchinfo>=1.8.0,<2',
'torchmetrics>=0.11.4,<1',
'speechbrain>=0.5.14,<1',
'audiomentations>=0.30.0,<1',
'torch-audiomentations>=0.11.0,<1',
'tqdm>=4.64.0,<5',
'pytest>=7.2.0,<8',
'pytest-cov>=2.10.1,<3',
'pytest-flake8>=1.1.1,<2',
'pytest-mypy>=0.10.0,<1',
'plotext>=5.2.7,<6',
'sounddevice>=0.4.1,<1'
'acoustics>=0.2.6,<1',
'pyyaml>=6.0,<7',
'tensorflow==2.8.1',
'tensorflow_probability==0.16.0',
'protobuf>=3.20,<4',
'onnx_tf==1.10.0',
'onnx==1.14.0',
'pronouncing>=0.2.0,<1',
'datasets>=2.14.4,<3',
'deep-phonemizer==0.0.19'
]
},
author="David Scripka",