diff --git a/README.md b/README.md index c65c557..75dac45 100644 --- a/README.md +++ b/README.md @@ -46,9 +46,23 @@ frame = my_function_to_get_audio_frame() prediction = model.predict(frame) ``` -## Reccomended Settings +# Reccomendations for Usage -While the default settings for openWakeWord will work well in many cases, there are adjustable parameters that can improve performance in some cases. On supported platforms (currently only X86 and Arm64 linux), Speex noise suppression can be enabled by setting the `enable_speex_noise_suppression=True` when instantiating an openWakeWord model. This can improve performance when relatively constant background noise is present. Second, a voice activity deteciton (VAD) model from [Silero](https://github.com/snakers4/silero-vad) is included with openWakeWord, and can be enabled by setting the `vad_threshold` argument to a value between 0 and 1 when instantiating an openWakeWord model. This will only allow a positive prediction from openWakeWord when the VAD model simultaneously has a score above the specified threshold, which can significantly reduce false-positive activations in the present of non-speech noise. Finally, all of the included openWakeWord models were trained to work well with a default threshold of `0.5` for a positive prediction, but you are encouraged to determine the best threshold for your environment and use-case through testing. +## Noise Suppresion and Voice Activity Detection (VAD) + +While the default settings for openWakeWord will work well in many cases, there are adjustable parameters in openWakeWord that can improve performance in some deployment scenarios. + +On supported platforms (currently only X86 and Arm64 linux), Speex noise suppression can be enabled by setting the `enable_speex_noise_suppression=True` when instantiating an openWakeWord model. This can improve performance when relatively constant background noise is present. + +Second, a voice activity deteciton (VAD) model from [Silero](https://github.com/snakers4/silero-vad) is included with openWakeWord, and can be enabled by setting the `vad_threshold` argument to a value between 0 and 1 when instantiating an openWakeWord model. This will only allow a positive prediction from openWakeWord when the VAD model simultaneously has a score above the specified threshold, which can significantly reduce false-positive activations in the present of non-speech noise. + +## Threshold Scores for Activation + +All of the included openWakeWord models were trained to work well with a default threshold of `0.5` for a positive prediction, but you are encouraged to determine the best threshold for your environment and use-case through testing. For certain deployments, using a lower or higher threshold in practice may result in significantly better performance. + +## User-specific models + +If the baseline performance of openWakeWord models is not sufficent for a given application (specifically, if the false activation rate is unacceptably high), it is possible to train [custom verifier models](docs/custom_verifier_models.md) for specific voices that act as a second-stage filter on predictions (i.e., only allow activations through that were likely spoken by a known set of voices). This can greatly improve performance, at the cost of making the openWakeWord system less likely to respond to voices new voices. # Project Goals diff --git a/docs/custom_verifier_models.md b/docs/custom_verifier_models.md new file mode 100644 index 0000000..19f4aec --- /dev/null +++ b/docs/custom_verifier_models.md @@ -0,0 +1,47 @@ +# Custom Verifier Models + +If the performance of a trained openWakeWord model is not sufficient in a production application, training a custom verifier model on a particular speaker or set of speakers can help significantly the performance of the system. A custom verify model acts as a filter on top of the base openWakeWord model, determining whether a given activation was likely from a known target speaker. In particular, this can be a very effective way at reducing false activiations, as the model will be more focused on a the target speaker instead of attempting to activate for any speaker. + +There are trade-offs to this approach, however. In general, training a custom verifier model can be beneficial with two assumptions: + +1) It is feasible to collect the training data required to build a custom model for all of the desired users of the system. The training requirements are minimal (likely <5 minutes of effort), but needs to be repeated for every user. + +2) The range of acoustic environments seen in production are similar enough to that observed during collection of the user-specific data. If there are singicant differences across deployment acoustic environments, custom models will need to be trained for each one. + +# Verifier Model Design + +The custom verifier models are designed to be very lightweight and easy to train. For the current version of openWakeWord, the verifier models are simple logistic regression binary classifiers the take in the shared audio features from the openWakeWord preprocessing stage and returns a score between 0 and 1 indicating whether the audio contains a wakeword or phrase spoken by the target speaker. Because this task in inherently much more narrow compared to the detecting the wakeword or phrase from any speaker, the combination of the verifier model and base model can be quite effective. + +Note that while the verifier model is focused on a target speaker, it is not intended to perform the task of speaker verification directly. Performance on this task may be adequate for certain use-case cases, but caution is recommended. + +# Verifier Model Training + +Training a custom verifier model is conceptually simple, and only requires a very small amount of training data. Reccomendations for training data collection are listed below. + +- Positive data (examples of wakeword or phrase) + - Collect a minimum of 3 examples for each target speaker + - Positive examples should be as close as possible to the expected deployment scenario, including some level of background noise if that is appropriate + +- Negative data collection + - Collect a minimum of ~10 seconds of speech from each target speaker that does not contain the wakword, trying to include as much variation as possible in the speech + - Optionally, collect ~5 seconds clips of typical background audio in the deployment evironment or use previously collected examples of false activations (this is one of the most effective ways to reduce false activations) + +After collected the positive and negative examples, a custom verifier model can be trained with the `openwakeword.train_custom_verifier` function: + +```python +openwakeword.train_custom_verifier( + positive_reference_clips = ["positive_clip1.wav", "positive_clip2.wav", "positive_clip3.wav"] + negative_reference_clips = ["negative_clip1.wav", "negative_clip2.wav"] + output_path = "path/to/directory/model.pkl" + model_name = "hey_jarvis.onnx" # the target model which matches the wake word/phrase of the collected positive examples +) +``` + +After training a model and saving it, an openWakeWord instance can be created with the verifier model which will be called whenever the base openWakeWord model makes a prediction with a score above the specified threshold, and the returned score will be the one from the verifier model. + +```python +oww = openwakeword.Model( + custom_verifier_models={"hey_jarvis": "path_to_verifier_model.pkl"}, + custom_verifier_threshold=0.3, # the threshold score required to invoke the verifier model +) +``` \ No newline at end of file diff --git a/openwakeword/__init__.py b/openwakeword/__init__.py index 1d88247..9dd2eaf 100755 --- a/openwakeword/__init__.py +++ b/openwakeword/__init__.py @@ -3,7 +3,7 @@ from openwakeword.model import Model from openwakeword.vad import VAD from openwakeword.custom_verifier_model import train_custom_verifier -__all__ = ['Model', 'VAD', train_custom_verifier] +__all__ = ['Model', 'VAD', 'train_custom_verifier'] models = { "alexa": { diff --git a/openwakeword/custom_verifier_model.py b/openwakeword/custom_verifier_model.py index b49ff9c..1616528 100644 --- a/openwakeword/custom_verifier_model.py +++ b/openwakeword/custom_verifier_model.py @@ -25,8 +25,8 @@ from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.preprocessing import FunctionTransformer, StandardScaler -# Define functions to prepare data for speaker dependent verifier model +# Define functions to prepare data for speaker dependent verifier model def get_reference_clip_features( reference_clip: str, oww_model: openwakeword.Model, @@ -34,11 +34,11 @@ def get_reference_clip_features( threshold: float = 0.5, N: int = 3, **kwargs - ): + ): """ Processes input audio files (16-bit, 16-khz single-channel WAV files) and gets the openWakeWord audio features that produce a prediction from the specified model greater than the threshold value. - + Args: reference_clip (str): The target audio file to get features from @@ -56,7 +56,7 @@ def get_reference_clip_features( # Create dictionary to store frames positive_data = collections.defaultdict(list) - + # Get predictions for _ in range(N): # Load clip @@ -64,27 +64,32 @@ def get_reference_clip_features( sr, dat = scipy.io.wavfile.read(reference_clip) else: dat = reference_clip - + # Set random starting point to get small variations in features if N != 1: - dat = dat[np.random.randint(0,1280):] - + dat = dat[np.random.randint(0, 1280):] + # Get predictions step_size = 1280 for i in range(0, dat.shape[0]-step_size, step_size): predictions = oww_model.predict(dat[i:i+step_size], **kwargs) if predictions[model_name] >= threshold: - features = oww_model.preprocessor.get_features(oww_model.model_inputs[model_name]) + features = oww_model.preprocessor.get_features( # type: ignore[has-type] + oww_model.model_inputs[model_name] # type: ignore[has-type] + ) positive_data[model_name].append(features) if len(positive_data[model_name]) == 0: - positive_data[model_name].append(np.empty((0, oww_model.model_inputs[model_name], 96))) - + positive_data[model_name].append( + np.empty((0, oww_model.model_inputs[model_name], 96))) # type: ignore[has-type] + return np.vstack(positive_data[model_name]) + def flatten_features(x): return [i.flatten() for i in x] + def train_verifier_model(features: np.ndarray, labels: np.ndarray): """ Train a logistic regression binary classifier model on the provided features and labels @@ -105,6 +110,7 @@ def train_verifier_model(features: np.ndarray, labels: np.ndarray): return pipeline + def train_custom_verifier( positive_reference_clips: str, negative_reference_clips: str, @@ -136,27 +142,27 @@ def train_custom_verifier( wakeword_model_paths=[model_name], **kwargs ) - model_name = model_name[0:-5] + model_name = model_name.split(os.path.sep)[-1][0:-5] else: oww = openwakeword.Model(**kwargs) # Get features from positive reference clips positive_features = np.vstack( - [get_reference_clip_features(i, oww, model_name, N=5) + [get_reference_clip_features(i, oww, model_name, N=5) for i in tqdm(positive_reference_clips, desc="Processing positive reference clips")] ) # Get features from negative reference clips negative_features = np.vstack( - [get_reference_clip_features(i, oww, model_name, threshold=0.0, N=1) + [get_reference_clip_features(i, oww, model_name, threshold=0.0, N=1) for i in tqdm(negative_reference_clips, desc="Processing negative reference clips")] ) - + # Train logistic regression model on reference clip features print("Training and saving verifier model...") lr_model = train_verifier_model( np.vstack((positive_features, negative_features)), - [1]*positive_features.shape[0] + [0]*negative_features.shape[0] + np.array([1]*positive_features.shape[0] + [0]*negative_features.shape[0]) ) # Save logistic regression model to specified output location diff --git a/openwakeword/model.py b/openwakeword/model.py index c349e22..72524ac 100755 --- a/openwakeword/model.py +++ b/openwakeword/model.py @@ -62,12 +62,14 @@ class Model(): For every input audio frame, a VAD score is obtained and only those model predictions with VAD scores above the threshold will be returned. The default value (0), disables voice activity detection entirely. - custom_verifier_models (dict): A dictionary of paths to custom verifier models, where - the keys are the model names (corresponding to the openwakeword.models attribute) - and the values are the filepaths of the custom verifier models. - custom_verifier_threshold (float): The score threshold to use a custom verifier model. If the score from a model for - a given frame is greater than this value, the associated custom verifier model will - also predict on that frame, and the verifier score will be returned. + custom_verifier_models (dict): A dictionary of paths to custom verifier models, where + the keys are the model names (corresponding to the openwakeword.models + attribute) and the values are the filepaths of the + custom verifier models. + custom_verifier_threshold (float): The score threshold to use a custom verifier model. If the score + from a model for a given frame is greater than this value, the + associated custom verifier model will also predict on that frame, and + the verifier score will be returned. kwargs (dict): Any other keyword arguments to pass the the preprocessor instance """ @@ -210,7 +212,7 @@ class Model(): self.preprocessor.get_features(self.model_inputs[mdl]) )[0][-1] predictions[cls] = verifier_prediction - + # Update prediction buffer, and zero predictions for first 5 frames during model initialization for cls in predictions.keys(): if len(self.prediction_buffer[cls]) < 5: diff --git a/setup.py b/setup.py index 87fea77..0af6ceb 100644 --- a/setup.py +++ b/setup.py @@ -27,7 +27,7 @@ def build_additional_requires(): setuptools.setup( name="openwakeword", version="0.2.0", - install_requires=['onnxruntime>=1.10.0,<2'], + install_requires=['onnxruntime>=1.10.0,<2', 'tqdm>=4.0,<5.0', 'scipy>=1.3,<2'], extras_require={ 'test': [ 'pytest>=7.2.0,<8', diff --git a/tests/test_custom_verifier_model.py b/tests/test_custom_verifier_model.py new file mode 100644 index 0000000..d9576b2 --- /dev/null +++ b/tests/test_custom_verifier_model.py @@ -0,0 +1,65 @@ +# Copyright 2022 David Scripka. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# Copyright 2022 David Scripka. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Imports +import openwakeword +import os +import numpy as np +import scipy.io.wavfile +import tempfile + + +# Tests +class TestModels: + def test_train_verifier_model(self): + with tempfile.TemporaryDirectory() as tmp_dir: + # Make random negative data for verifier model training + scipy.io.wavfile.write(os.path.join(tmp_dir, "negative_reference.wav"), + 16000, np.random.randint(-1000, 1000, 16000*4).astype(np.int16)) + + # Load random clips + reference_clips = [os.path.join("tests", "data", "hey_mycroft_test.wav")] + negative_clips = [os.path.join(tmp_dir, "negative_reference.wav")] + + # Train verifier model on the reference clips + openwakeword.train_custom_verifier( + positive_reference_clips=reference_clips, + negative_reference_clips=negative_clips, + output_path=os.path.join(tmp_dir, 'verifier_model.pkl'), + model_name="hey_mycroft" + ) + + # Load model with verifier model + owwModel = openwakeword.Model( + wakeword_model_paths=[os.path.join("openwakeword", "resources", "models", "hey_mycroft_v0.1.onnx")], + custom_verifier_models={"hey_mycroft_v0.1": os.path.join(tmp_dir, "verifier_model.pkl")}, + custom_verifier_threshold=0.3, + ) + + # Prediction on random data + owwModel.predict_clip(reference_clips[0])