akadmin/openwakeword
1
0
Fork 0
mirror of https://github.com/dscripka/openWakeWord.git synced 2026-08-27 18:17:20 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #17 from dscripka/custom_verifier_model

Browse source 
Custom verifier model
This commit is contained in:
dscripka 2023-02-17 22:17:11 -05:00 • committed by GitHub
commit d5fd0e954d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 332 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

18
README.md
View file

@ -46,9 +46,23 @@ frame = my_function_to_get_audio_frame()
prediction = model.predict(frame)
```
## Reccomended Settings
# Reccomendations for Usage
While the default settings for openWakeWord will work well in many cases, there are adjustable parameters that can improve performance in some cases. On supported platforms (currently only X86 and Arm64 linux), Speex noise suppression can be enabled by setting the `enable_speex_noise_suppression=True` when instantiating an openWakeWord model. This can improve performance when relatively constant background noise is present. Second, a voice activity deteciton (VAD) model from [Silero](https://github.com/snakers4/silero-vad) is included with openWakeWord, and can be enabled by setting the `vad_threshold` argument to a value between 0 and 1 when instantiating an openWakeWord model. This will only allow a positive prediction from openWakeWord when the VAD model simultaneously has a score above the specified threshold, which can significantly reduce false-positive activations in the present of non-speech noise. Finally, all of the included openWakeWord models were trained to work well with a default threshold of `0.5` for a positive prediction, but you are encouraged to determine the best threshold for your environment and use-case through testing.
## Noise Suppresion and Voice Activity Detection (VAD)
While the default settings for openWakeWord will work well in many cases, there are adjustable parameters in openWakeWord that can improve performance in some deployment scenarios.
On supported platforms (currently only X86 and Arm64 linux), Speex noise suppression can be enabled by setting the `enable_speex_noise_suppression=True` when instantiating an openWakeWord model. This can improve performance when relatively constant background noise is present.
Second, a voice activity deteciton (VAD) model from [Silero](https://github.com/snakers4/silero-vad) is included with openWakeWord, and can be enabled by setting the `vad_threshold` argument to a value between 0 and 1 when instantiating an openWakeWord model. This will only allow a positive prediction from openWakeWord when the VAD model simultaneously has a score above the specified threshold, which can significantly reduce false-positive activations in the present of non-speech noise.
## Threshold Scores for Activation
All of the included openWakeWord models were trained to work well with a default threshold of `0.5` for a positive prediction, but you are encouraged to determine the best threshold for your environment and use-case through testing. For certain deployments, using a lower or higher threshold in practice may result in significantly better performance.
## User-specific models
If the baseline performance of openWakeWord models is not sufficent for a given application (specifically, if the false activation rate is unacceptably high), it is possible to train [custom verifier models](docs/custom_verifier_models.md) for specific voices that act as a second-stage filter on predictions (i.e., only allow activations through that were likely spoken by a known set of voices). This can greatly improve performance, at the cost of making the openWakeWord system less likely to respond to voices new voices.
# Project Goals

47
docs/custom_verifier_models.md Normal file
View file

@ -0,0 +1,47 @@
# Custom Verifier Models
If the performance of a trained openWakeWord model is not sufficient in a production application, training a custom verifier model on a particular speaker or set of speakers can help significantly the performance of the system. A custom verify model acts as a filter on top of the base openWakeWord model, determining whether a given activation was likely from a known target speaker. In particular, this can be a very effective way at reducing false activiations, as the model will be more focused on a the target speaker instead of attempting to activate for any speaker.
There are trade-offs to this approach, however. In general, training a custom verifier model can be beneficial with two assumptions:
1) It is feasible to collect the training data required to build a custom model for all of the desired users of the system. The training requirements are minimal (likely <5 minutes of effort), but needs to be repeated for every user.
2) The range of acoustic environments seen in production are similar enough to that observed during collection of the user-specific data. If there are singicant differences across deployment acoustic environments, custom models will need to be trained for each one.
# Verifier Model Design
The custom verifier models are designed to be very lightweight and easy to train. For the current version of openWakeWord, the verifier models are simple logistic regression binary classifiers the take in the shared audio features from the openWakeWord preprocessing stage and returns a score between 0 and 1 indicating whether the audio contains a wakeword or phrase spoken by the target speaker. Because this task in inherently much more narrow compared to the detecting the wakeword or phrase from any speaker, the combination of the verifier model and base model can be quite effective.
Note that while the verifier model is focused on a target speaker, it is not intended to perform the task of speaker verification directly. Performance on this task may be adequate for certain use-case cases, but caution is recommended.
# Verifier Model Training
Training a custom verifier model is conceptually simple, and only requires a very small amount of training data. Reccomendations for training data collection are listed below.
- Positive data (examples of wakeword or phrase)
- Collect a minimum of 3 examples for each target speaker
- Positive examples should be as close as possible to the expected deployment scenario, including some level of background noise if that is appropriate
- Negative data collection
- Collect a minimum of ~10 seconds of speech from each target speaker that does not contain the wakword, trying to include as much variation as possible in the speech
- Optionally, collect ~5 seconds clips of typical background audio in the deployment evironment or use previously collected examples of false activations (this is one of the most effective ways to reduce false activations)
After collected the positive and negative examples, a custom verifier model can be trained with the `openwakeword.train_custom_verifier` function:
```python
openwakeword.train_custom_verifier(
positive_reference_clips = ["positive_clip1.wav", "positive_clip2.wav", "positive_clip3.wav"]
negative_reference_clips = ["negative_clip1.wav", "negative_clip2.wav"]
output_path = "path/to/directory/model.pkl"
model_name = "hey_jarvis.onnx" # the target model which matches the wake word/phrase of the collected positive examples
)
```
After training a model and saving it, an openWakeWord instance can be created with the verifier model which will be called whenever the base openWakeWord model makes a prediction with a score above the specified threshold, and the returned score will be the one from the verifier model.
```python
oww = openwakeword.Model(
custom_verifier_models={"hey_jarvis": "path_to_verifier_model.pkl"},
custom_verifier_threshold=0.3, # the threshold score required to invoke the verifier model
)
```

3
openwakeword/__init__.py
View file

@ -1,8 +1,9 @@
import os
from openwakeword.model import Model
from openwakeword.vad import VAD
from openwakeword.custom_verifier_model import train_custom_verifier
__all__ = ['Model', 'VAD']
__all__ = ['Model', 'VAD', 'train_custom_verifier']
models = {
"alexa": {

170
openwakeword/custom_verifier_model.py Normal file
View file

@ -0,0 +1,170 @@
# Copyright 2022 David Scripka. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Imports
import os
from tqdm import tqdm
import collections
import openwakeword
import numpy as np
import scipy
import pickle
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler
# Define functions to prepare data for speaker dependent verifier model
def get_reference_clip_features(
reference_clip: str,
oww_model: openwakeword.Model,
model_name: str,
threshold: float = 0.5,
N: int = 3,
**kwargs
):
"""
Processes input audio files (16-bit, 16-khz single-channel WAV files) and gets the openWakeWord
audio features that produce a prediction from the specified model greater than the threshold value.
Args:
reference_clip (str): The target audio file to get features from
oww_model (openwakeword.Model): The openWakeWord model object used to get predictions
model_name (str): The name of the model to get predictions from (should correspond to
a python dictionary key in the oww_model.models attribute)
threshold (float): The minimum score from the model required to capture the associated features
N (int): How many times to run feature extraction for a given clip, adding some slight variation
in the starting position each time to ensure that the features are not identical
Returns:
ndarray: A numpy array of shape N x M x L, where N is the number of examples, M is the number
of frames in the window, and L is the audio feature/embedding dimension.
"""
# Create dictionary to store frames
positive_data = collections.defaultdict(list)
# Get predictions
for _ in range(N):
# Load clip
if type(reference_clip) == str:
sr, dat = scipy.io.wavfile.read(reference_clip)
else:
dat = reference_clip
# Set random starting point to get small variations in features
if N != 1:
dat = dat[np.random.randint(0, 1280):]
# Get predictions
step_size = 1280
for i in range(0, dat.shape[0]-step_size, step_size):
predictions = oww_model.predict(dat[i:i+step_size], **kwargs)
if predictions[model_name] >= threshold:
features = oww_model.preprocessor.get_features( # type: ignore[has-type]
oww_model.model_inputs[model_name] # type: ignore[has-type]
)
positive_data[model_name].append(features)
if len(positive_data[model_name]) == 0:
positive_data[model_name].append(
np.empty((0, oww_model.model_inputs[model_name], 96))) # type: ignore[has-type]
return np.vstack(positive_data[model_name])
def flatten_features(x):
return [i.flatten() for i in x]
def train_verifier_model(features: np.ndarray, labels: np.ndarray):
"""
Train a logistic regression binary classifier model on the provided features and labels
Args:
features (ndarray): A N x M numpy array, where N is the number of examples and M
is the number of features
labels (ndarray): A 1D numpy array where each value corresponds to the label of the Nth
example in the `features` argument
Returns:
The trained scikit-learn logistic regression model
"""
# C value matters alot here, depending on dataset size (larger datasets work better with larger C?)
clf = LogisticRegression(random_state=0, max_iter=2000, C=0.001)
pipeline = make_pipeline(FunctionTransformer(flatten_features), StandardScaler(), clf)
pipeline.fit(features, labels)
return pipeline
def train_custom_verifier(
positive_reference_clips: str,
negative_reference_clips: str,
output_path: str,
model_name: str,
**kwargs
):
"""
Trains a voice-specific custom verifier model on examples of wake word/phrase speech and other speech
from a single user.
Args:
positive_reference_clips (str): The path to a directory containing single-channel 16khz, 16-bit WAV files
of the target wake word/phrase.
negative_reference_clips (str): The path to a directory containing single-channel 16khz, 16-bit WAV files
of miscellaneous speech not containing the target wake word/phrase.
output_path (str): The location to save the trained verifier model (as a scikit-learn .joblib file)
model_name (str): The name or path of the trained openWakeWord model that the verifier model will be
based on. If only a name, it must be one of the pre-trained models included in the
openWakeWord release.
kwargs: Any other keyword arguments to pass to the openWakeWord model initialization
Returns:
None
"""
# Load target openWakeWord model
if os.path.exists(model_name):
oww = openwakeword.Model(
wakeword_model_paths=[model_name],
**kwargs
)
model_name = model_name.split(os.path.sep)[-1][0:-5]
else:
oww = openwakeword.Model(**kwargs)
# Get features from positive reference clips
positive_features = np.vstack(
[get_reference_clip_features(i, oww, model_name, N=5)
for i in tqdm(positive_reference_clips, desc="Processing positive reference clips")]
)
# Get features from negative reference clips
negative_features = np.vstack(
[get_reference_clip_features(i, oww, model_name, threshold=0.0, N=1)
for i in tqdm(negative_reference_clips, desc="Processing negative reference clips")]
)
# Train logistic regression model on reference clip features
print("Training and saving verifier model...")
lr_model = train_verifier_model(
np.vstack((positive_features, negative_features)),
np.array([1]*positive_features.shape[0] + [0]*negative_features.shape[0])
)
# Save logistic regression model to specified output location
print("Done!")
pickle.dump(lr_model, open(output_path, "wb"))

32
openwakeword/model.py
View file

@ -20,6 +20,7 @@ from openwakeword.utils import AudioFeatures
import wave
import os
import pickle
from collections import deque, defaultdict
from functools import partial
import time
@ -38,6 +39,8 @@ class Model():
class_mapping_dicts: List[dict] = [],
enable_speex_noise_suppression: bool = False,
vad_threshold: float = 0,
custom_verifier_models: Union[bool, dict] = False,
custom_verifier_threshold: float = 0.1,
**kwargs
):
"""Initialize the openWakeWord model object.
@ -59,6 +62,15 @@ class Model():
For every input audio frame, a VAD score is obtained and only those model predictions
with VAD scores above the threshold will be returned. The default value (0),
disables voice activity detection entirely.
custom_verifier_models (dict): A dictionary of paths to custom verifier models, where
the keys are the model names (corresponding to the openwakeword.models
attribute) and the values are the filepaths of the
custom verifier models.
custom_verifier_threshold (float): The score threshold to use a custom verifier model. If the score
from a model for a given frame is greater than this value, the
associated custom verifier model will also predict on that frame, and
the verifier score will be returned.
kwargs (dict): Any other keyword arguments to pass the the preprocessor instance
"""
# Initialize the ONNX models and store them
@ -73,13 +85,16 @@ class Model():
else:
wakeword_model_names = [os.path.basename(i[0:-5]) for i in wakeword_model_paths]
# Create attributes to store models and metadat
# Create attributes to store models and metadata
self.models = {}
self.model_inputs = {}
self.model_outputs = {}
self.class_mapping = {}
self.model_input_names = {}
self.custom_verifier_models = {}
self.custom_verifier_threshold = custom_verifier_threshold
for mdl_path, mdl_name in zip(wakeword_model_paths, wakeword_model_names):
# Load openwakeword models
self.models[mdl_name] = ort.InferenceSession(mdl_path, sess_options=sessionOptions,
providers=["CPUExecutionProvider"])
self.model_inputs[mdl_name] = self.models[mdl_name].get_inputs()[0].shape[1]
@ -92,6 +107,11 @@ class Model():
self.class_mapping[mdl_name] = {str(i): str(i) for i in range(0, self.model_outputs[mdl_name])}
self.model_input_names[mdl_name] = self.models[mdl_name].get_inputs()[0].name
# Load custom verifier models
if isinstance(custom_verifier_models, dict):
if custom_verifier_models.get(mdl_name, False):
self.custom_verifier_models[mdl_name] = pickle.load(open(custom_verifier_models[mdl_name], 'rb'))
# Create buffer to store frame predictions
self.prediction_buffer: DefaultDict[str, deque] = defaultdict(partial(deque, maxlen=30))
@ -183,6 +203,16 @@ class Model():
for int_label, cls in self.class_mapping[mdl].items():
predictions[cls] = prediction[0][0][int(int_label)]
# Update scores based on custom verifier model
if self.custom_verifier_models != {}:
for cls in predictions.keys():
if predictions[cls] >= self.custom_verifier_threshold:
parent_model = self.get_parent_model_from_label(cls)
verifier_prediction = self.custom_verifier_models[parent_model].predict_proba(
self.preprocessor.get_features(self.model_inputs[mdl])
)[0][-1]
predictions[cls] = verifier_prediction
# Update prediction buffer, and zero predictions for first 5 frames during model initialization
for cls in predictions.keys():
if len(self.prediction_buffer[cls]) < 5:

2
setup.py
View file

@ -27,7 +27,7 @@ def build_additional_requires():
setuptools.setup(
name="openwakeword",
version="0.2.0",
install_requires=['onnxruntime>=1.10.0,<2'],
install_requires=['onnxruntime>=1.10.0,<2', 'tqdm>=4.0,<5.0', 'scipy>=1.3,<2', 'scikit-learn>=1,<2'],
extras_require={
'test': [
'pytest>=7.2.0,<8',

65
tests/test_custom_verifier_model.py Normal file
View file

@ -0,0 +1,65 @@
# Copyright 2022 David Scripka. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 David Scripka. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Imports
import openwakeword
import os
import numpy as np
import scipy.io.wavfile
import tempfile
# Tests
class TestModels:
def test_train_verifier_model(self):
with tempfile.TemporaryDirectory() as tmp_dir:
# Make random negative data for verifier model training
scipy.io.wavfile.write(os.path.join(tmp_dir, "negative_reference.wav"),
16000, np.random.randint(-1000, 1000, 16000*4).astype(np.int16))
# Load random clips
reference_clips = [os.path.join("tests", "data", "hey_mycroft_test.wav")]
negative_clips = [os.path.join(tmp_dir, "negative_reference.wav")]
# Train verifier model on the reference clips
openwakeword.train_custom_verifier(
positive_reference_clips=reference_clips,
negative_reference_clips=negative_clips,
output_path=os.path.join(tmp_dir, 'verifier_model.pkl'),
model_name="hey_mycroft"
)
# Load model with verifier model
owwModel = openwakeword.Model(
wakeword_model_paths=[os.path.join("openwakeword", "resources", "models", "hey_mycroft_v0.1.onnx")],
custom_verifier_models={"hey_mycroft_v0.1": os.path.join(tmp_dir, "verifier_model.pkl")},
custom_verifier_threshold=0.3,
)
# Prediction on random data
owwModel.predict_clip(reference_clips[0])