{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "79d90b54",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook demonstrates how the performance metrics for openWakeWord models are calculated, specifically the false-reject and false-accept rates based on testing datasets of WAV files.\n",
    "\n",
    "Note that calculating these metrics often involves some degree of subjectivity. In particular, choices around how to count false-reject and false-accepts can have a large impact on the resulting metrics. For openWakeWord, the following choices were made:\n",
    "\n",
    "- False-reject counts are based on individual clips that contain a *single* positive example. If the score for any frame in that clip is above a threshold, it counts as a true-accept for that clip.\n",
    "- False-accept counts are also based on counts of frame scores above a threshold. *However*, to avoid double-counting, some basic filtering is applied to the time-series of frame scores to ensure that each false activation is only counted once. See the `openwakeword.metrics.get_false_positives` function for more details, but in short, after a single frame above the threshold the subsequent 50 frames (4 seconds) are ignored before another false activation will be counted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "43df2824",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T23:38:38.099943Z",
     "start_time": "2023-01-01T23:38:37.720170Z"
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "\n",
    "import os\n",
    "import platform\n",
    "\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "import IPython.display as ipd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import openwakeword\n",
    "from openwakeword.utils import bulk_predict\n",
    "from openwakeword.metrics import get_false_positives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36f0ca63",
   "metadata": {},
   "source": [
    "# Download Dataset(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a804a549",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T20:15:56.812113Z",
     "start_time": "2023-01-01T20:15:37.368533Z"
    }
   },
   "outputs": [],
   "source": [
    "# Download and extract DipCo corpus (warning! This is 12GB)\n",
    "\n",
    "if platform.system() == \"Linux\":\n",
    "    !wget https://s3.amazonaws.com/dipco/DiPCo.tgz\n",
    "else:\n",
    "    !curl -O https://s3.amazonaws.com/dipco/DiPCo.tgz\n",
    "\n",
    "!tar -zxf DipCo.tgz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7897f159",
   "metadata": {},
   "source": [
    "#### Links for some useful testing data\n",
    "\n",
    "Picovoice \"alexa\" dataset: https://github.com/Picovoice/wake-word-benchmark/tree/master/audio/alexa\n",
    "\n",
    "Note that some files (like the \"alexa\" dataset) will need to be converted to 16 khz, 16-bit wav files.\n",
    "You can use the `openwakeword.data.convert_clips` utility function to do so, which requires either `sox` or `ffmpeg` as the system-level backend."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86d4457a",
   "metadata": {},
   "source": [
    "# Load Trained openWakeWord Model(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "489b4aa5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T23:39:26.668594Z",
     "start_time": "2023-01-01T23:39:26.475270Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/dscripka/anaconda3/envs/torch_gpu/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Load a trained openWakeWord models\n",
    "\n",
    "model_path = \"../openwakeword/resources/models/alexa_v0.1.onnx\"\n",
    "model_name = \"alexa_v0.1\"\n",
    "oww = openwakeword.Model(\n",
    "    wakeword_model_paths=[model_path]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af898ac7",
   "metadata": {},
   "source": [
    "# Estimate False accept rate/false positive rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "643e7362",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T23:41:54.836368Z",
     "start_time": "2023-01-01T23:39:30.353574Z"
    }
   },
   "outputs": [],
   "source": [
    "# Get paths to test files (DipCo corpus)\n",
    "\n",
    "clips = [str(i) for i in Path(\"DiPCo/audio/\").glob(\"**/*U01.CH1.wav\")]\n",
    "\n",
    "# Do bulk prediction in parrallel for faster inference\n",
    "predictions = bulk_predict(\n",
    "    file_paths = clips,\n",
    "    wakeword_model_paths=[model_path],\n",
    "    ncpu=10 # set this to the number of CPU cores you have available\n",
    ")\n",
    "\n",
    "# Flatten predictions into a single list, as we don't need to differentiate between files for this test\n",
    "predictions_dipco = [i[model_name] for j in predictions.keys() for i in predictions[j]]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "69805d78",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T23:48:23.855870Z",
     "start_time": "2023-01-01T23:48:23.815887Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate the number of false positives\n",
    "\n",
    "num_false_positives = get_false_positives(predictions_dipco, threshold=0.5, grouping_window=50)\n",
    "num_false_positives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f1615f5b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T23:48:26.138814Z",
     "start_time": "2023-01-01T23:48:25.980125Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 1.0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 44560 characters]\n",
      "text/plain": [
       "<Figure size 1600x900 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot results\n",
    "plt.figure(figsize=(16,9))\n",
    "plt.plot(predictions_dipco)\n",
    "plt.ylim(0,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25440876",
   "metadata": {},
   "source": [
    "# False reject rate/false negative rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "38f59da6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T23:48:49.561065Z",
     "start_time": "2023-01-01T23:48:31.593726Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False Reject Rate @0.5 threshold: 0.030395136778115502\n"
     ]
    }
   ],
   "source": [
    "# Get paths to test files (positive examples)\n",
    "\n",
    "clips = [str(i) for i in Path(\"alexa_test_clips/\").glob(\"**/*.wav\")] # assumes these clips are 16khz, 16-bit WAV files\n",
    "\n",
    "# Do bulk prediction for faster testing\n",
    "predictions_frr = bulk_predict(\n",
    "    file_paths = clips,\n",
    "    wakeword_model_paths=[model_path],\n",
    "    ncpu=10,\n",
    "    padding=2 # pad each clip with n seconds of silence before & after\n",
    ")\n",
    "\n",
    "# Count how many files had a prediction above a given threshold\n",
    "threshold = 0.5\n",
    "fn = 0\n",
    "fn_clips = []\n",
    "positive_scores = []\n",
    "for fname, scores in predictions_frr.items():\n",
    "    scores = [i[model_name] for i in scores]\n",
    "    positive_scores.append(max(scores))\n",
    "    if positive_scores[-1] < threshold:\n",
    "        fn += 1\n",
    "        fn_clips.append(fname)\n",
    "\n",
    "print(f\"False Reject Rate @{threshold} threshold: {fn/len(clips)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49da97d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T17:05:23.733771Z",
     "start_time": "2023-01-01T17:05:23.708960Z"
    }
   },
   "outputs": [],
   "source": [
    "# (optionally) listen to the falsely rejected clips\n",
    "\n",
    "for i in fn_clips:\n",
    "    sr, dat = scipy.io.wavfile.read(i)\n",
    "    ipd.display(ipd.Audio(dat, rate=16000, normalize=True, autoplay=False))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba2880d",
   "metadata": {},
   "source": [
    "# Plot False-Acccept/False-Reject Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d4e77328",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-01-01T23:49:52.585060Z",
     "start_time": "2023-01-01T23:49:51.021926Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [00:01<00:00, 33.60it/s]\n",
      "100%|██████████| 50/50 [00:00<00:00, 41734.37it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'False Reject Rate')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 34820 characters]\n",
      "text/plain": [
       "<Figure size 1600x900 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compute and plot the false-accept/false-reject curve from the metrics above (requires that all cells have been run)\n",
    "\n",
    "from openwakeword.metrics import generate_roc_curve_fprs, generate_roc_curve_tprs\n",
    "\n",
    "fprs = generate_roc_curve_fprs(predictions_dipco, n_points=50)\n",
    "tprs = generate_roc_curve_tprs(positive_scores, n_points=50)\n",
    "frr = [1 - i for i in tprs]\n",
    "\n",
    "# Make plot\n",
    "plt.figure(figsize=(16,9))\n",
    "plt.plot(fprs, frr)\n",
    "plt.ylim(0,max(frr)*2)\n",
    "plt.xlim(0,2)\n",
    "plt.xlabel(\"False Accepts/Hour\")\n",
    "plt.ylabel(\"False Reject Rate\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2207997e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch_gpu",
   "language": "python",
   "name": "torch_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "228.667px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
