{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "241df33e",
   "metadata": {},
   "source": [
    "# Simple keyword spotting with CMSIS-DSP Python wrapper and Arduino"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3652bb1c",
   "metadata": {},
   "source": [
    "The goal of this notebook is to demonstrate how to use the CMSIS-DSP Python wrapper on an example which is complex enough.\n",
    "\n",
    "It is not a state of the art keyword recognition system. The feature used for the machine learning is very simple and just able to recognize the \"Yes\" keyword.\n",
    "\n",
    "But it is a good start and enough to demonstrate lots of features of the Python wrapper like:\n",
    "\n",
    "* Testing the CMSIS-DSP algorithm directly in Python\n",
    "* Test of fixed point implementation\n",
    "* Implementation of the compute graph and streaming computation from the CMSIS-DSP Synchronous Data Flow framework\n",
    "* C++ code generation for the compute graph\n",
    "* Final implementation for Arduino Nano 33 BLE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23aa0adb",
   "metadata": {},
   "source": [
    "Several Python packages are required. If they are not already installed on you system, you can install them from the notebook by using:\n",
    "\n",
    "`!pip install packagename`\n",
    "\n",
    "The machine learning is using scikit-learn. For scientific computations, we are using SciPy, NumPy and Matplotlib.\n",
    "\n",
    "For reading wav files, the soundfile package is used.\n",
    "\n",
    "Other packages will be used below in the notebook and will have to be installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d8ef4050",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import cmsisdsp as dsp\n",
    "import cmsisdsp.fixedpoint as fix\n",
    "import numpy as np\n",
    "import os.path\n",
    "import glob\n",
    "import pathlib\n",
    "import random\n",
    "import soundfile as sf\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import display,Audio,HTML\n",
    "import scipy.signal\n",
    "from numpy.lib.stride_tricks import sliding_window_view\n",
    "from scipy.signal.windows import hann\n",
    "from sklearn import svm\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import uniform\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b7926c9",
   "metadata": {},
   "source": [
    "## The speech commands\n",
    "\n",
    "We are using the simplified speech commands from the TensorFlow Lite tutorial.\n",
    "\n",
    "Those commands can be downloaded from this link: \"http://storage.googleapis.com/download.tensorflow.org/data/mini_speech_commands.zip\"\n",
    "\n",
    "Once the zip has been uncompressed, you'll need to change the path of the folder below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "352220f9",
   "metadata": {},
   "source": [
    "The below code is loading the list of commands available in the `mini_speech_commands` folder and it is describing the words we want to detect. Here we only want to detect the `Yes` keyword.\n",
    "\n",
    "You can add other keywords but the CMSIS-DSP implementation in this notebook is only supporting one keyword.\n",
    "Nevertheless, if you'd like to experiment with the training of the ML model and different features, then you can work with more commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3fb202b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "MINISPEECH=\"mini_speech_commands\"\n",
    "commands=np.array([os.path.basename(f) for f in glob.glob(os.path.join(MINISPEECH,\"mini_speech_commands\",\"*\"))])\n",
    "commands=commands[commands != \"README.md\"]\n",
    "# Any other word will be recognized as unknown\n",
    "to_keep=['yes']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc4b610",
   "metadata": {},
   "source": [
    "The below code is generating a label ID for a command. The ID will be -1 for any command not in the `to_keep` list. Other ID will be the index of the keyword in this list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9df05672",
   "metadata": {},
   "outputs": [],
   "source": [
    "UNKNOWN_CLASS = -1\n",
    "def get_label(name):\n",
    "    return(pathlib.PurePath(name).parts[-2])\n",
    "def get_label_id(name):\n",
    "    label=get_label(name)\n",
    "    if label in to_keep:\n",
    "        return(to_keep.index(label))\n",
    "    else:\n",
    "        return(UNKNOWN_CLASS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cf6b745",
   "metadata": {},
   "source": [
    "## The feature\n",
    "\n",
    "The feature is based on a simple zero crossing rate (zcr). We choose to only keep the increasing crossing. I don't think it is making a lot of differences for the final performance of the keyword recognition.\n",
    "\n",
    "The `zcr` function is computing the zcr for a window of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f24e5b4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def zcr(w):\n",
    "    w = w-np.mean(w)\n",
    "    f=w[:-1]\n",
    "    g=w[1:]\n",
    "    k=np.count_nonzero(np.logical_and(f*g<0, g>f))\n",
    "    return(1.0*k/len(f))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a218c0e5",
   "metadata": {},
   "source": [
    "The final feature is the zcr computed on a segment of 1 second and filtered. We are using a sliding window and using a Hann window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4dc76d14",
   "metadata": {},
   "outputs": [],
   "source": [
    "def feature(data):\n",
    "    samplerate=16000\n",
    "    input_len = 16000\n",
    "    \n",
    "    # The speech pattern is padded to ensure it has a duration of 1 second\n",
    "    \n",
    "    waveform = data[:input_len]\n",
    "    \n",
    "    zero_padding = np.zeros(\n",
    "      16000 - waveform.shape[0],\n",
    "      dtype=np.float32)\n",
    "   \n",
    "   \n",
    "    signal = np.hstack([waveform, zero_padding])\n",
    "    \n",
    "    \n",
    "    # We decompose the intput signal into overlapping window. And the signal in each window\n",
    "    # is premultiplied by a Hann window of the right size.\n",
    "    # Warning : if you change the window duration and audio offset, you'll need to change the value\n",
    "    # in the scripts used for the scheduling of the compute graph later.\n",
    "    winDuration=25e-3\n",
    "    audioOffsetDuration=10e-3\n",
    "    winLength=int(np.floor(samplerate*winDuration))\n",
    "    audioOffset=int(np.floor(samplerate*audioOffsetDuration))\n",
    "    overlap=winLength-audioOffset\n",
    "    window=hann(winLength,sym=False)\n",
    "    reta=[zcr(x*window) for x in sliding_window_view(signal,winLength)[::audioOffset,:]]\n",
    "    \n",
    "    # The final signal is filtered. We have tested several variations on the feature. This filtering is\n",
    "    # improving the recognition\n",
    "    reta=scipy.signal.lfilter(np.ones(10)/10.0,[1],reta)\n",
    "    return(np.array(reta))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40b040df",
   "metadata": {},
   "source": [
    "## The patterns\n",
    "\n",
    "The below class is representing a Pattern. A pattern can either be a sound file from the TensorFlow Lite examples and in this case we compute the feature and the label id.\n",
    "\n",
    "The pattern can also be a random white noise. In that case, we also compute the feature and the class id is -1.\n",
    "\n",
    "Note that when you use the signal property, the speech patterns will return the content of the file but noise patterns will generate a random noise which will thus be different each time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "279b0999",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Pattern:\n",
    "    def __init__(self,p):\n",
    "        global UNKNOWN_CLASS\n",
    "        if isinstance(p, str):\n",
    "            self._isFile=True \n",
    "            self._filename=p\n",
    "            self._label=get_label_id(p)\n",
    "    \n",
    "            data, samplerate = sf.read(self._filename)\n",
    "            self._feature = feature(data)\n",
    "        else:\n",
    "            self._isFile=False\n",
    "            self._noiseLevel=p\n",
    "            self._label=UNKNOWN_CLASS\n",
    "            \n",
    "            noise=np.random.randn(16000)*p\n",
    "            self._feature=feature(noise)\n",
    "            \n",
    "    @property\n",
    "    def label(self):\n",
    "        return(self._label)\n",
    "            \n",
    "    @property\n",
    "    def feature(self):\n",
    "        return(self._feature)\n",
    "        \n",
    "        \n",
    "    # Only useful for plotting\n",
    "    # The random pattern will be different each time\n",
    "    @property\n",
    "    def signal(self):\n",
    "        if not self._isFile:\n",
    "            return(np.random.randn(16000)*self._noiseLevel)\n",
    "        else:\n",
    "            data, samplerate = sf.read(self._filename)\n",
    "            return(data)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "099433f6",
   "metadata": {},
   "source": [
    "Following code is giving the number of speech samples for each keyword.\n",
    "It is assuming that all keywords contain the same number of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "79588da0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files_per_command=len(glob.glob(os.path.join(MINISPEECH,\"mini_speech_commands\",commands[0],\"*\")))\n",
    "files_per_command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb329acb",
   "metadata": {},
   "source": [
    "The following code is generating the patterns used for the training of the ML model,.\n",
    "It is reading patterns for all the words we want to keep (from `to_keep` list) and it is aggregating all other keywords in the unknown class.\n",
    "\n",
    "It is also generating some random noise patterns.For the unknown class, the number of patterns will always be `files_per_command` but some patterns may be noise rather than sound files.\n",
    "\n",
    "There is some randomization of file names. So each time this code is executed, you'll get patterns in a different order and for the unknown class, which is containing more than `files_per_command`, you'll get a different subset of those patterns (thr subset will have the right length `files_per_command`).\n",
    "\n",
    "Finally the patterns are also randomized so that the split between training and test patterns will select different patterns each time this code is executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e4e8bf7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add patterns we want to detect\n",
    "filenames=[]\n",
    "for f in to_keep:\n",
    "    filenames+=glob.glob(os.path.join(MINISPEECH,\"mini_speech_commands\",f,\"*\"))\n",
    "\n",
    "    \n",
    "random.shuffle(filenames)\n",
    "    \n",
    "# Add remaining patterns\n",
    "remaining_words=list(set(commands)-set(to_keep))\n",
    "nb_noise=0\n",
    "\n",
    "remaining=[]\n",
    "for f in remaining_words:\n",
    "    remaining+=glob.glob(os.path.join(MINISPEECH,\"mini_speech_commands\",f,\"*\"))\n",
    "    \n",
    "random.shuffle(remaining)\n",
    "\n",
    "\n",
    "filenames += remaining[0:files_per_command-nb_noise]\n",
    "\n",
    "patterns=[Pattern(x) for x in filenames]\n",
    "\n",
    "for i in range(nb_noise):\n",
    "    patterns.append(Pattern(np.abs(np.random.rand(1)*0.05)[0]))\n",
    "    \n",
    "random.shuffle(patterns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc8b0033",
   "metadata": {},
   "source": [
    "Below code is extracting the training and test patterns.\n",
    "This will be used later to generate the array used by scikit learn to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e9ebb293",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000\n"
     ]
    }
   ],
   "source": [
    "print(len(patterns))\n",
    "patterns=np.array(patterns)\n",
    "\n",
    "nb_patterns = len(patterns)\n",
    "nb_train= int(np.floor(0.8 * nb_patterns))\n",
    "nb_tests=nb_patterns-nb_train\n",
    "\n",
    "train_patterns = patterns[:nb_train]\n",
    "test_patterns = patterns[-nb_tests:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bab9b1d2",
   "metadata": {},
   "source": [
    "## Testing on a signal\n",
    "\n",
    "The following code is displaying a pattern as example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b298cfea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15408 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "nbpat=50\n",
    "data = patterns[nbpat].signal\n",
    "samplerate=16000\n",
    "plt.plot(data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bbdb9edd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "audio=Audio(data=data,rate=samplerate,autoplay=False)\n",
    "audio"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "460afa2a",
   "metadata": {},
   "source": [
    "Simple function to display a spectrogram. It is adapted from a SciPy example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8cd9c83f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_spectrogram(waveform,fs):\n",
    "    # Zero-padding for an audio waveform with less than 16,000 samples.\n",
    "    input_len = 16000\n",
    "    waveform = waveform[:input_len]\n",
    "    zero_padding = np.zeros(\n",
    "      16000 - waveform.shape[0],\n",
    "      dtype=np.float32)\n",
    "    mmax=np.max(np.abs(waveform))\n",
    "    \n",
    "    equal_length = np.hstack([waveform, zero_padding])\n",
    "    f, t, Zxx = scipy.signal.stft(equal_length, fs, nperseg=1000)\n",
    "    plt.pcolormesh(t, f, np.abs(Zxx), vmin=0, vmax=mmax/100, shading='gouraud')\n",
    "    plt.title('STFT Magnitude')\n",
    "    plt.ylabel('Frequency [Hz]')\n",
    "    plt.xlabel('Time [sec]')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "92eca740",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 118052 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "get_spectrogram(data,16000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03037e1b",
   "metadata": {},
   "source": [
    "Display of the feature to compare with the spectrogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c55f0d03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15452 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "feat=feature(data)\n",
    "plt.plot(feat)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b798aea",
   "metadata": {},
   "source": [
    "## Patterns for training\n",
    "\n",
    "Now we generate the arrays needed to train and test the model.\n",
    "We have an array of feature : X_array.\n",
    "An array of label ID : y\n",
    "\n",
    "and similar arrays for the tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "647ecb32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1600, 98)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X=np.array([x.feature for x in train_patterns])\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2603ecb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1600,)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y=np.array([x.label for x in train_patterns])\n",
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "658b7553",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = [x.label for x in test_patterns]\n",
    "X_test = [x.feature for x in test_patterns]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79909063",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n",
    "We have chosen to use a simple logistic regression. We are doing a randomized search on the hyperparameter space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e1fdf0ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "distributionsb = dict(C=uniform(loc=1, scale=1000)\n",
    "                    )\n",
    "reg = LogisticRegression(penalty=\"l1\", solver=\"saga\", tol=0.1)\n",
    "clfb=RandomizedSearchCV(reg, distributionsb,random_state=0,n_iter=50).fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bca2dabd",
   "metadata": {},
   "source": [
    "We are using the best estimator found during the randomized search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fddd7782",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=682.8202991034834, penalty='l1', solver='saga', tol=0.1)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clfb.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a7ffeb1",
   "metadata": {},
   "source": [
    "The confusion matrix is generated from the test patterns to check the behavior of the classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "eb6facde",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x2457342cb80>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 10528 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = clfb.predict(X_test)\n",
    "labels=[\"Unknown\"] + to_keep\n",
    "ConfusionMatrixDisplay.from_predictions(y_test, y_pred,display_labels=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b17c7d",
   "metadata": {},
   "source": [
    "We compute the final score. 0.8 is really the minimum acceptable value for this kind of demo.\n",
    "With the zcr feature, if you try to detect `Yes`, `No`, `Unknown`, you'll get a score of around 0.6 which is very bad."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "de8460b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.83"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clfb.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba6739f",
   "metadata": {},
   "source": [
    "We can now save the model so that next time we want to play with the notebook and test the CMSIS-DSP implementation we do not have to retrain the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ea4ae8f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"logistic.pickle\",\"wb\") as f:\n",
    "    s = pickle.dump(clfb,f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aab3aaf2",
   "metadata": {},
   "source": [
    "And we can reload the saved model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "id": "c4811945",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"logistic.pickle\",\"rb\") as f:\n",
    "    clfb=pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9c1ef1b",
   "metadata": {},
   "source": [
    "## Reference implementation with Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef865bc5",
   "metadata": {},
   "source": [
    "This is the reference implementation which will be used to build the CMSIS-DSP implementation. We are no more using the scikit-learn predict function instead we are using an implementation of predict using linear algebra. It should give the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f6e861cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(feat):\n",
    "    coef=clfb.best_estimator_.coef_\n",
    "    intercept=clfb.best_estimator_.intercept_\n",
    "    \n",
    "    res=np.dot(coef,feat) + intercept\n",
    "    \n",
    "    if res<0:\n",
    "        return(-1)\n",
    "    else:\n",
    "        return(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "918afd57",
   "metadata": {},
   "source": [
    "And like in the code above with scikit-learn, we are checking the result with the confusion matrix and the score. It should give the same results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "66e69345",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x24575807190>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 10528 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred_ref = [predict(x) for x in X_test]\n",
    "labels=[\"Unknown\"] + to_keep\n",
    "ConfusionMatrixDisplay.from_predictions(y_test, y_pred_ref,display_labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "887d714a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.83"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.count_nonzero(np.equal(y_test,y_pred_ref))/len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bbd0d3b",
   "metadata": {},
   "source": [
    "## CMSIS-DSP implementation\n",
    "\n",
    "Now we are ready to implement the code using CMSIS-DSP API. Once we have a running implementation in Python, writing the C code will be easy since the API is the same.\n",
    "\n",
    "We are testing 3 implementations here : F32, Q31 and Q15.\n",
    "At the end, we will check that Q15 is giving good enough results and thus can be implemented in C for the Arduino."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a801479c",
   "metadata": {},
   "source": [
    "### F32 Implementation\n",
    "\n",
    "It will be very similar to the implemenattion above with matrix but will instead use the CMSIS-DSP API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d861ee1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "coef_f32=clfb.best_estimator_.coef_\n",
    "intercept_f32=clfb.best_estimator_.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1e50c5b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_zcr(w):\n",
    "    m = dsp.arm_mean_f32(w)\n",
    "    m = -m\n",
    "    w = dsp.arm_offset_f32(w,m)\n",
    "    f=w[:-1]\n",
    "    g=w[1:]\n",
    "    k=np.count_nonzero(np.logical_and(f*g<0, g>f))\n",
    "    return(1.0*k/len(f))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa0b092d",
   "metadata": {},
   "source": [
    "For the FIR, CMSIS-DSP is using a FIR instance structure and thus we need to define it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "3313b384",
   "metadata": {},
   "outputs": [],
   "source": [
    "firf32 = dsp.arm_fir_instance_f32()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "480631ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_feature(data):\n",
    "    samplerate=16000\n",
    "    input_len = 16000\n",
    "    \n",
    "    waveform = data[:input_len]\n",
    "    \n",
    "    zero_padding = np.zeros(\n",
    "      16000 - waveform.shape[0],\n",
    "      dtype=np.float32)\n",
    "   \n",
    "   \n",
    "    signal = np.hstack([waveform, zero_padding])\n",
    "    \n",
    "    \n",
    "    winDuration=25e-3\n",
    "    audioOffsetDuration=10e-3\n",
    "    winLength=int(np.floor(samplerate*winDuration))\n",
    "    audioOffset=int(np.floor(samplerate*audioOffsetDuration))\n",
    "    overlap=winLength -audioOffset\n",
    "    window=hann(winLength,sym=False)\n",
    "    reta=[dsp_zcr(dsp.arm_mult_f32(x,window)) for x in sliding_window_view(signal,winLength)[::audioOffset,:]]\n",
    "    \n",
    "    # Reset state and filter\n",
    "    # We want to start with a clean filter each time we filter a new feature.\n",
    "    # So the filter state is reset each time.\n",
    "    blockSize=98\n",
    "    numTaps=10\n",
    "    stateLength = numTaps + blockSize - 1\n",
    "    dsp.arm_fir_init_f32(firf32,10,np.ones(10)/10.0,np.zeros(stateLength))\n",
    "    reta=dsp.arm_fir_f32(firf32,reta)\n",
    "    return(np.array(reta))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d683590",
   "metadata": {},
   "source": [
    "Let's check that the feature is giving the same result as the reference implemenattion using linear algebra."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "1b926a1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15452 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "feat=dsp_feature(data)\n",
    "plt.plot(feat)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f894f1f0",
   "metadata": {},
   "source": [
    "The feature code is working, so now we can implement the predict:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "a67a5b5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_predict(feat):\n",
    "    \n",
    "    res=dsp.arm_dot_prod_f32(coef_f32,feat)\n",
    "    res = res + intercept_f32\n",
    "    \n",
    "    if res[0]<0:\n",
    "        return(-1)\n",
    "    else:\n",
    "        return(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fccb1c17",
   "metadata": {},
   "source": [
    "And finally we can check the CMSIS-DSP behavior of the test patterns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "84bac863",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x2457340b6d0>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 10528 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred_ref = [dsp_predict(dsp_feature(x.signal)) for x in test_patterns]\n",
    "labels=[\"Unknown\"] + to_keep\n",
    "ConfusionMatrixDisplay.from_predictions(y_test, y_pred_ref,display_labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "bf0a23d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.83"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.count_nonzero(np.equal(y_test,y_pred_ref))/len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a80b698",
   "metadata": {},
   "source": [
    "We are getting very similar results to the reference implementation. Now let's explore fixed point."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3bcc892",
   "metadata": {},
   "source": [
    "### Q31 implementation\n",
    "\n",
    "First thing to do is to convert the F32 values of the ML mode into Q31.\n",
    "But we need values in [-1,1]. So we rescale those values and keep track of the shift required to restore the original value.\n",
    "\n",
    "Then, we convert those rescaled values to Q31."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "beb7e364",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_coef=clfb.best_estimator_.coef_ \n",
    "coef_shift=0\n",
    "while np.max(np.abs(scaled_coef)) > 1:\n",
    "    scaled_coef = scaled_coef / 2.0 \n",
    "    coef_shift = coef_shift + 1\n",
    "\n",
    "coef_q31=fix.toQ31(scaled_coef)\n",
    "\n",
    "scaled_intercept = clfb.best_estimator_.intercept_ \n",
    "intercept_shift = 0\n",
    "while np.abs(scaled_intercept) > 1:\n",
    "    scaled_intercept = scaled_intercept / 2.0 \n",
    "    intercept_shift = intercept_shift + 1\n",
    "    \n",
    "intercept_q31=fix.toQ31(scaled_intercept)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1cbb2c",
   "metadata": {},
   "source": [
    "Now we can implement the zcr and feature in Q31."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "7a64829c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_zcr_q31(w):\n",
    "    m = dsp.arm_mean_q31(w)\n",
    "    # Negate can saturate so we use CMSIS-DSP function which is working on array (and we have a scalar)\n",
    "    m = dsp.arm_negate_q31(np.array([m]))[0]\n",
    "    w = dsp.arm_offset_q31(w,m)\n",
    "    \n",
    "    f=w[:-1]\n",
    "    g=w[1:]\n",
    "    k=np.count_nonzero(np.logical_and(np.logical_or(np.logical_and(f>0,g<0), np.logical_and(f<0,g>0)),g>f))\n",
    "    \n",
    "    # k < len(f) so shift should be 0 except when k == len(f)\n",
    "    # When k==len(f) normally quotient is 0x40000000 and shift 1 and we convert\n",
    "    # this to 0x7FFFFFF and shift 0\n",
    "    status,quotient,shift_val=dsp.arm_divide_q31(k,len(f))\n",
    "    if shift_val==1:\n",
    "        return(dsp.arm_shift_q31(np.array([quotient]),shift)[0])\n",
    "    else:\n",
    "        return(quotient)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2aac9a63",
   "metadata": {},
   "outputs": [],
   "source": [
    "firq31 = dsp.arm_fir_instance_q31()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a65003af",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_feature_q31(data):\n",
    "    samplerate=16000\n",
    "    input_len = 16000\n",
    "    \n",
    "    waveform = data[:input_len]\n",
    "    \n",
    "    zero_padding = np.zeros(\n",
    "      16000 - waveform.shape[0],\n",
    "      dtype=np.int32)\n",
    "   \n",
    "   \n",
    "    signal = np.hstack([waveform, zero_padding])\n",
    "    \n",
    "    \n",
    "    winDuration=25e-3\n",
    "    audioOffsetDuration=10e-3\n",
    "    winLength=int(np.floor(samplerate*winDuration))\n",
    "    audioOffset=int(np.floor(samplerate*audioOffsetDuration))\n",
    "    overlap=winLength-audioOffset\n",
    "    \n",
    "    window=fix.toQ31(hann(winLength,sym=False))\n",
    "    reta=[dsp_zcr_q31(dsp.arm_mult_q31(x,window)) for x in sliding_window_view(signal,winLength)[::audioOffset,:]]\n",
    "    \n",
    "    # Reset state and filter\n",
    "    blockSize=98\n",
    "    numTaps=10\n",
    "    stateLength = numTaps + blockSize - 1\n",
    "    dsp.arm_fir_init_q31(firq31,10,fix.toQ31(np.ones(10)/10.0),np.zeros(stateLength,dtype=np.int32))\n",
    "    reta=dsp.arm_fir_q31(firq31,reta)\n",
    "    return(np.array(reta))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e63b0ceb",
   "metadata": {},
   "source": [
    "Let's check the feature on the data to compare with the F32 version and check it is working:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "d59c2442",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15452 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "feat=fix.Q31toF32(dsp_feature_q31(fix.toQ31(data)))\n",
    "plt.plot(feat)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccd39f92",
   "metadata": {},
   "source": [
    "The Q31 feature is very similar to the F32 one so now we can implement the predict:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "0df93150",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_predict_q31(feat):\n",
    "    \n",
    "    res=dsp.arm_dot_prod_q31(coef_q31,feat)\n",
    "    \n",
    "    # Before adding the res and the intercept we need to ensure they are in the same Qx.x format\n",
    "    # The scaling applied to the coefs and to the intercept is different so we need to scale\n",
    "    # the intercept to take this into account\n",
    "    scaled=dsp.arm_shift_q31(np.array([intercept_q31]),intercept_shift-coef_shift)[0]\n",
    "    # Because dot prod output is in Q16.48\n",
    "    # and ret is on 64 bits\n",
    "    scaled = np.int64(scaled) << 17 \n",
    "    \n",
    "    res = res + scaled\n",
    "    \n",
    "   \n",
    "    \n",
    "    if res<0:\n",
    "        return(-1)\n",
    "    else:\n",
    "        return(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8691cb89",
   "metadata": {},
   "source": [
    "Now we can check the Q31 implementation on the test patterns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "7f8787ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x245733de5f0>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 10824 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred_ref = [dsp_predict_q31(dsp_feature_q31(fix.toQ31(x.signal))) for x in test_patterns]\n",
    "labels=[\"Unknown\"] + to_keep\n",
    "ConfusionMatrixDisplay.from_predictions(y_test, y_pred_ref,display_labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "500da0b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8225"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.count_nonzero(np.equal(y_test,y_pred_ref))/len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4cb5d66",
   "metadata": {},
   "source": [
    "The score is as good as the F32 implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22e5033e",
   "metadata": {},
   "source": [
    "### Q15 Implementation\n",
    "\n",
    "It is the same as Q31 but using Q15 functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "cf7674a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_coef=clfb.best_estimator_.coef_ \n",
    "coef_shift=0\n",
    "while np.max(np.abs(scaled_coef)) > 1:\n",
    "    scaled_coef = scaled_coef / 2.0 \n",
    "    coef_shift = coef_shift + 1\n",
    "\n",
    "coef_q15=fix.toQ15(scaled_coef)\n",
    "\n",
    "scaled_intercept = clfb.best_estimator_.intercept_ \n",
    "intercept_shift = 0\n",
    "while np.abs(scaled_intercept) > 1:\n",
    "    scaled_intercept = scaled_intercept / 2.0 \n",
    "    intercept_shift = intercept_shift + 1\n",
    "    \n",
    "intercept_q15=fix.toQ15(scaled_intercept)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "3e2258ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_zcr_q15(w):\n",
    "    m = dsp.arm_mean_q15(w)\n",
    "    # Negate can saturate so we use CMSIS-DSP function which is working on array (and we have a scalar)\n",
    "    m = dsp.arm_negate_q15(np.array([m]))[0]\n",
    "    w = dsp.arm_offset_q15(w,m)\n",
    "    \n",
    "    f=w[:-1]\n",
    "    g=w[1:]\n",
    "    k=np.count_nonzero(np.logical_and(np.logical_or(np.logical_and(f>0,g<0), np.logical_and(f<0,g>0)),g>f))\n",
    "    \n",
    "    # k < len(f) so shift should be 0 except when k == len(f)\n",
    "    # When k==len(f) normally quotient is 0x4000 and shift 1 and we convert\n",
    "    # this to 0x7FFF and shift 0\n",
    "    status,quotient,shift_val=dsp.arm_divide_q15(k,len(f))\n",
    "    if shift_val==1:\n",
    "        return(dsp.arm_shift_q15(np.array([quotient]),shift)[0])\n",
    "    else:\n",
    "        return(quotient)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "345c8f73",
   "metadata": {},
   "outputs": [],
   "source": [
    "firq15 = dsp.arm_fir_instance_q15()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6a974f45",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_feature_q15(data):\n",
    "    samplerate=16000\n",
    "    input_len = 16000\n",
    "    \n",
    "    waveform = data[:input_len]\n",
    "    \n",
    "    zero_padding = np.zeros(\n",
    "      16000 - waveform.shape[0],\n",
    "      dtype=np.int16)\n",
    "   \n",
    "   \n",
    "    signal = np.hstack([waveform, zero_padding])\n",
    "    \n",
    "    \n",
    "    winDuration=25e-3\n",
    "    audioOffsetDuration=10e-3\n",
    "    winLength=int(np.floor(samplerate*winDuration))\n",
    "    audioOffset=int(np.floor(samplerate*audioOffsetDuration))\n",
    "    overlap=winLength - audioOffset\n",
    "    \n",
    "    window=fix.toQ15(hann(winLength,sym=False))\n",
    "    reta=[dsp_zcr_q15(dsp.arm_mult_q15(x,window)) for x in sliding_window_view(signal,winLength)[::audioOffset,:]]\n",
    "    \n",
    "    # Reset state and filter\n",
    "    blockSize=98\n",
    "    numTaps=10\n",
    "    stateLength = numTaps + blockSize - 1\n",
    "    dsp.arm_fir_init_q15(firq15,10,fix.toQ15(np.ones(10)/10.0),np.zeros(stateLength,dtype=np.int16))\n",
    "    reta=dsp.arm_fir_q15(firq15,reta)\n",
    "    return(np.array(reta))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "18a33368",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15516 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "feat=fix.Q15toF32(dsp_feature_q15(fix.toQ15(data)))\n",
    "plt.plot(feat)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "9ad998ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dsp_predict_q15(feat):\n",
    "    \n",
    "    res=dsp.arm_dot_prod_q15(coef_q15,feat)\n",
    "    \n",
    "    scaled=dsp.arm_shift_q15(np.array([intercept_q15]),intercept_shift-coef_shift)[0]\n",
    "    # Because dot prod output is in Q34.30\n",
    "    # and ret is on 64 bits\n",
    "    scaled = np.int64(scaled) << 15 \n",
    "    \n",
    "    res = res + scaled\n",
    "    \n",
    "   \n",
    "    \n",
    "    if res<0:\n",
    "        return(-1)\n",
    "    else:\n",
    "        return(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "501be275",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x24575b54bb0>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 10792 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred_ref = [dsp_predict_q15(dsp_feature_q15(fix.toQ15(x.signal))) for x in test_patterns]\n",
    "labels=[\"Unknown\"] + to_keep\n",
    "ConfusionMatrixDisplay.from_predictions(y_test, y_pred_ref,display_labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "511f5d36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8175"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.count_nonzero(np.equal(y_test,y_pred_ref))/len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c1ce084",
   "metadata": {},
   "source": [
    "Q15 version is as good as other versions so we are selecting this implementation to run on the Arduino (once it has been converted to C)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c3fed6a",
   "metadata": {},
   "source": [
    "## Synchronous Data Flow\n",
    "\n",
    "We are receiving stream of samples but our functions are using buffers.\n",
    "We have sliding windows which make it more difficult to connect those functions together : we need FIFOs.\n",
    "\n",
    "So we are going to use the CMSIS-DSP Synchronous Data Flow framework to describe the compute graph, compute the FIFO lengths and generate a static schedule implementing the streaming computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "e17fecbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cmsisdsp.sdf.scheduler import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b273e79c",
   "metadata": {},
   "source": [
    "To describe our compute graph, we need to describe the nodes which are used in this graph.\n",
    "Each node is described by its inputs and outputs. For each IO, we define the data type and the number of samples read or written on the IO.\n",
    "\n",
    "We need the following nodes in our system:\n",
    "* A source to get audio samples from the Arduino PDM driver\n",
    "* A Sink to generate the messages on the Arduino serial port\n",
    "* A feature node to compute the feature for a given window (and pre-multipliying with the Hann window)\n",
    "* A FIR node which is filtering all the features for one second of signal\n",
    "* A KWS node which is doing the logistic regression\n",
    "\n",
    "In addition to that we need sliding windows:\n",
    "* Like in the Python code above we need a sliding window for audio\n",
    "* After each recognition attempt on a segment of 1 second, we want to slide the recognition window by 0.5 seconds. It can be implemented with a sliding window on the features before the FIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "6dad5c54",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Source(GenericSource):\n",
    "    def __init__(self,name,inLength):\n",
    "        GenericSource.__init__(self,name)\n",
    "        q15Type=CType(Q15)\n",
    "        self.addOutput(\"o\",q15Type,inLength)\n",
    "\n",
    "    @property\n",
    "    def typeName(self):\n",
    "        return \"Source\"\n",
    "    \n",
    "class Sink(GenericSink):\n",
    "    def __init__(self,name,outLength):\n",
    "        GenericSink.__init__(self,name)\n",
    "        q15Type=CType(Q15)\n",
    "        self.addInput(\"i\",q15Type,outLength)\n",
    "\n",
    "    @property\n",
    "    def typeName(self):\n",
    "        return \"Sink\"\n",
    "    \n",
    "class Feature(GenericNode):\n",
    "    def __init__(self,name,inLength):\n",
    "        GenericNode.__init__(self,name)\n",
    "\n",
    "        q15Type=CType(Q15)\n",
    "        self.addInput(\"i\",q15Type,inLength)\n",
    "        self.addOutput(\"o\",q15Type,1)\n",
    "\n",
    "    @property\n",
    "    def typeName(self):\n",
    "        return \"Feature\"\n",
    "    \n",
    "class FIR(GenericNode):\n",
    "    def __init__(self,name,inLength,outLength):\n",
    "        GenericNode.__init__(self,name)\n",
    "\n",
    "        q15Type=CType(Q15)\n",
    "        self.addInput(\"i\",q15Type,inLength)\n",
    "        self.addOutput(\"o\",q15Type,outLength)\n",
    "\n",
    "    @property\n",
    "    def typeName(self):\n",
    "        return \"FIR\"\n",
    "    \n",
    "class KWS(GenericNode):\n",
    "    def __init__(self,name,inLength):\n",
    "        GenericNode.__init__(self,name)\n",
    "\n",
    "        q15Type=CType(Q15)\n",
    "        self.addInput(\"i\",q15Type,inLength)\n",
    "        self.addOutput(\"o\",q15Type,1)\n",
    "\n",
    "    @property\n",
    "    def typeName(self):\n",
    "        return \"KWS\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a07cf93a",
   "metadata": {},
   "source": [
    "We need some parameters. Those parameters need to be coherent with the values defined in the features in the above code.\n",
    "AUDIO_INTERRUPT_LENGTH is the audio length generated by the source. But it is not the audio length generated by th PDM driver on Arduino. The Arduino implementation of the Source is doing the adaptation as we will see below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "c37f2dbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "q15Type=CType(Q15)\n",
    "FS=16000\n",
    "winDuration=25e-3\n",
    "audioOffsetDuration=10e-3\n",
    "winLength=int(np.floor(FS*winDuration))\n",
    "audio_input_length=int(np.floor(FS*audioOffsetDuration))\n",
    "AUDIO_INTERRUPT_LENGTH = audio_input_length"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa4ed03b",
   "metadata": {},
   "source": [
    "Below function is :\n",
    "* Defining the compute graph by connecting all the nodes\n",
    "* Generating a Python implementation of the compute graph and its scheduling\n",
    "* Generating a C++ implementation of the compute graph and its scheduling\n",
    "* Generating a graphviz description of the graph\n",
    "\n",
    "The feature length is hardcoded. So if the sliding window parameters are changed, you'll need to change the values for the FEATURE_LENGTH and FEATURE_OVERLAP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "055749cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_sched(python_code=True):\n",
    "    src=Source(\"src\",AUDIO_INTERRUPT_LENGTH)\n",
    "    # For Python code, the input is a numpy array which is passed\n",
    "    # as argument of the node\n",
    "    if python_code:\n",
    "        src.addVariableArg(\"input_array\")\n",
    "    sink=Sink(\"sink\",1)\n",
    "\n",
    "    feature=Feature(\"feature\",winLength)\n",
    "    feature.addVariableArg(\"window\")\n",
    "\n",
    "    sliding_audio=SlidingBuffer(\"audioWin\",q15Type,winLength,winLength-audio_input_length)\n",
    "\n",
    "\n",
    "    FEATURE_LENGTH=98 # for one second\n",
    "    FEATURE_OVERLAP = 49 # We slide feature by 0.5 second\n",
    "    sliding_feature=SlidingBuffer(\"featureWin\",q15Type,FEATURE_LENGTH,FEATURE_OVERLAP)\n",
    "    kws=KWS(\"kws\",FEATURE_LENGTH)\n",
    "    # Parameters of the ML model used by the node.\n",
    "    kws.addVariableArg(\"coef_q15\")\n",
    "    kws.addVariableArg(\"coef_shift\")\n",
    "    kws.addVariableArg(\"intercept_q15\")\n",
    "    kws.addVariableArg(\"intercept_shift\")\n",
    "\n",
    "    fir=FIR(\"fir\",FEATURE_LENGTH,FEATURE_LENGTH)\n",
    "\n",
    "\n",
    "\n",
    "    # Description of the compute graph\n",
    "    g = Graph()\n",
    "\n",
    "    g.connect(src.o, sliding_audio.i)\n",
    "    g.connect(sliding_audio.o, feature.i)\n",
    "    g.connect(feature.o, sliding_feature.i)\n",
    "    g.connect(sliding_feature.o, fir.i)\n",
    "    g.connect(fir.o, kws.i)\n",
    "    g.connect(kws.o, sink.i)\n",
    "\n",
    "  \n",
    "    # For Python we run for only around 13 seconds of input signal.\n",
    "    # Without this, it would run forever.\n",
    "    conf=Configuration()\n",
    "    if python_code:\n",
    "        conf.debugLimit=13\n",
    "    \n",
    "    # We compute the scheduling\n",
    "    sched = g.computeSchedule(conf)\n",
    "\n",
    "    print(\"Schedule length = %d\" % sched.scheduleLength)\n",
    "    print(\"Memory usage %d bytes\" % sched.memory)\n",
    "\n",
    "    # We generate the scheduling code for a Python and C++ implementations\n",
    "    if python_code:\n",
    "        conf.pyOptionalArgs=\"input_array,window,coef_q15,coef_shift,intercept_q15,intercept_shift\"\n",
    "        sched.pythoncode(\".\",config=conf)\n",
    "        with open(\"test.dot\",\"w\") as f:\n",
    "             sched.graphviz(f)\n",
    "    else:\n",
    "        conf.cOptionalArgs=\"\"\"const q15_t *window,\n",
    "        const q15_t *coef_q15,\n",
    "        const int coef_shift,\n",
    "        const q15_t intercept_q15,\n",
    "        const int intercept_shift\"\"\"\n",
    "        conf.memoryOptimization=True\n",
    "        # When schedule is long\n",
    "        conf.codeArray=True\n",
    "        sched.ccode(\"kws\",config=conf)\n",
    "        with open(\"kws/test.dot\",\"w\") as f:\n",
    "             sched.graphviz(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b80da54",
   "metadata": {},
   "source": [
    "Next line is generating `sched.py` which is the Python implementation of the compute graph and its static scheduling. This file is describing the FIFOs connecting the nodes and describing how the nodes are scheduled.\n",
    "\n",
    "You still need to provide an implementation of the nodes. It is available in `appnodes.py` and it is nearly a copy/paste of the Q15 implementation above.\n",
    "\n",
    "But it is simpler because the sliding window and the static schedule ensure that each node is run only when enough data is available. So a big part of the control logic has been removed from the nodes.\n",
    "\n",
    "`sched.py` is long because the static schedule is long and there are lots of function calls.. When we generate the C++ implementation, we are using an option which is using an array to describe the static schedule. It makes the C++ code much shorter. But having the sequence of function calls can be useful for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "b87a56ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Schedule length = 151\n",
      "Memory usage 1612 bytes\n"
     ]
    }
   ],
   "source": [
    "gen_sched(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f283fc12",
   "metadata": {},
   "source": [
    "Next line is generating the C++ schedule that we will need for the Arduino implementation : `kws/scheduler.cpp`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "4875861f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Schedule length = 151\n",
      "Memory usage 1612 bytes\n"
     ]
    }
   ],
   "source": [
    "gen_sched(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8cc15da",
   "metadata": {},
   "source": [
    "Now we'd like to test the Q15 classifier and the static schedule on a real patterns.\n",
    "We are using the `Yes/No` pattern from our `VHT-SystemModeling` example.\n",
    "Below code is loading the pattern into a NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "6f50ba6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "import io\n",
    "import soundfile as sf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "47083054",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pattern_url=\"https://github.com/ARM-software/VHT-SystemModeling/blob/main/EchoCanceller/sounds/yesno.wav?raw=true\"\n",
    "f = urlopen(test_pattern_url)\n",
    "filedata = f.read()\n",
    "data, samplerate = sf.read(io.BytesIO(filedata))\n",
    "if len(data.shape)>1:\n",
    "    data=data[:,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c63c7749",
   "metadata": {},
   "source": [
    "Let's plot the signal to check we have the right one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "188670c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 13564 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3409e71e",
   "metadata": {},
   "source": [
    "Now we can run our static schedule on this file.\n",
    "The reload function are needed when debugging (or implementing) the `appnodes.py`. Without this, the package would not be reloaded in the notebook.\n",
    "\n",
    "This code needs some variables evaluated in the Q15 estimator code above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "a3a624a0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Yes\n",
      "Unknown\n",
      "Yes\n",
      "Yes\n",
      "Yes\n",
      "Unknown\n",
      "Yes\n",
      "Yes\n",
      "Unknown\n",
      "Yes\n",
      "Unknown\n",
      "Yes\n",
      "Unknown\n"
     ]
    }
   ],
   "source": [
    "import sched as s \n",
    "from importlib import reload\n",
    "import appnodes \n",
    "\n",
    "appnodes= reload(appnodes)\n",
    "\n",
    "s = reload(s)\n",
    "\n",
    "dataQ15=fix.toQ15(data)\n",
    "windowQ15=fix.toQ15(hann(winLength,sym=False))\n",
    "\n",
    "nb,error = s.scheduler(dataQ15,windowQ15,coef_q15,coef_shift,intercept_q15,intercept_shift)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65d58b64",
   "metadata": {},
   "source": [
    "The code is working. We are getting more printed `Yes` than `Yes` in the pattern because we are sliding by 0.5 second between each recognition and the same word can be recognized several time.\n",
    "\n",
    "Now we are ready to implement the same on an Arduino.\n",
    "First we need to generate the parameters of the model. If you have saved the model, you can reload it with the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "ee7c58d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"logistic.pickle\",\"rb\") as f:\n",
    "    clfb=pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e600613d",
   "metadata": {},
   "source": [
    "Once the model is loaded, we extract the values and convert them to Q15:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "cc3999d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_coef=clfb.best_estimator_.coef_ \n",
    "coef_shift=0\n",
    "while np.max(np.abs(scaled_coef)) > 1:\n",
    "    scaled_coef = scaled_coef / 2.0 \n",
    "    coef_shift = coef_shift + 1\n",
    "\n",
    "coef_q15=fix.toQ15(scaled_coef)\n",
    "\n",
    "scaled_intercept = clfb.best_estimator_.intercept_ \n",
    "intercept_shift = 0\n",
    "while np.abs(scaled_intercept) > 1:\n",
    "    scaled_intercept = scaled_intercept / 2.0 \n",
    "    intercept_shift = intercept_shift + 1\n",
    "    \n",
    "intercept_q15=fix.toQ15(scaled_intercept)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d35d8b49",
   "metadata": {},
   "source": [
    "Now we need to generate C arrays for the ML model parameters. Those parameters are generated into `kws/coef.cpp`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "65618fee",
   "metadata": {},
   "outputs": [],
   "source": [
    "def carray(a):\n",
    "    s=\"{\"\n",
    "    k=0 \n",
    "    for x in a:\n",
    "        s = s + (\"%d,\" % (x,))\n",
    "        k = k + 1 \n",
    "        if k == 10:\n",
    "            k=0;\n",
    "            s = s + \"\\n\"\n",
    "    s = s + \"}\"\n",
    "    return(s)\n",
    "\n",
    "ccode=\"\"\"#include \"arm_math.h\"\n",
    "#include \"coef.h\"\n",
    "\n",
    "\n",
    "const q15_t fir_coefs[NUMTAPS]=%s;\n",
    "\n",
    "\n",
    "const q15_t coef_q15[%d]=%s;\n",
    "\n",
    "const q15_t intercept_q15 = %d;\n",
    "const int coef_shift=%d;\n",
    "const int intercept_shift=%d;\n",
    " \n",
    "const q15_t window[%d]=%s;\n",
    "\"\"\"\n",
    "\n",
    "def gen_coef_code():\n",
    "    fir_coef = carray(fix.toQ15(np.ones(10)/10.0))\n",
    "    winq15=carray(fix.toQ15(hann(winLength,sym=False)))\n",
    "    res = ccode % (fir_coef,\n",
    "                   len(coef_q15[0]),\n",
    "                   carray(coef_q15[0]),\n",
    "                   intercept_q15,\n",
    "                   coef_shift,\n",
    "                   intercept_shift,\n",
    "                   winLength,\n",
    "                   winq15\n",
    "                  )\n",
    "    \n",
    "    with open(os.path.join(\"kws\",\"coef.cpp\"),\"w\") as f:\n",
    "        print(res,file=f)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "742406d2",
   "metadata": {},
   "source": [
    "Generation of the coef code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "2ee643d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "gen_coef_code()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7763f7",
   "metadata": {},
   "source": [
    "The implementation of the nodes is in `kws/AppNodes.h`. It is very similar to the `appnodes.py` but using the CMSIS-DSP C API.\n",
    "\n",
    "The C++ template are used only to minimize the overhead at runtime.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c92158e",
   "metadata": {},
   "source": [
    "## Arduino"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e419afd",
   "metadata": {},
   "source": [
    "You need to have the arduino command line tools installed. And they need to be in your `PATH` so that the notebook can find the tool.\n",
    "We are using the Arduino Nano 33 BLE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02896ac3",
   "metadata": {},
   "source": [
    "### Building and upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "41e89b2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Port Protocol Type              Board Name          FQBN                        Core             \n",
      "COM3 serial   Serial Port       Unknown                                                          \n",
      "COM6 serial   Serial Port (USB) Arduino Nano 33 BLE arduino:mbed_nano:nano33ble arduino:mbed_nano\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!arduino-cli board list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "b44dbf60",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Config file already exists, use --overwrite to discard the existing one.\n"
     ]
    }
   ],
   "source": [
    "!arduino-cli config init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "06b3c20e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading Arduino_CMSIS-DSP@5.7.0...\n",
      "Arduino_CMSIS-DSP@5.7.0 already downloaded\n",
      "Installing Arduino_CMSIS-DSP@5.7.0...\n",
      "Already installed Arduino_CMSIS-DSP@5.7.0\n"
     ]
    }
   ],
   "source": [
    "!arduino-cli lib install Arduino_CMSIS-DSP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f5909ca",
   "metadata": {},
   "source": [
    "The first time the below command is executed, it will take a __very__ long time. The full CMSIS-DSP library has to be rebuilt for the Arduino."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "0ea363fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sketch uses 98272 bytes (9%) of program storage space. Maximum is 983040 bytes.\n",
      "Global variables use 46760 bytes (17%) of dynamic memory, leaving 215384 bytes for local variables. Maximum is 262144 bytes.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!arduino-cli compile -b arduino:mbed_nano:nano33ble kws"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "bd13af3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device       : nRF52840-QIAA\n",
      "Version      : Arduino Bootloader (SAM-BA extended) 2.0 [Arduino:IKXYZ]\n",
      "Address      : 0x0\n",
      "Pages        : 256\n",
      "Page Size    : 4096 bytes\n",
      "Total Size   : 1024KB\n",
      "Planes       : 1\n",
      "Lock Regions : 0\n",
      "Locked       : none\n",
      "Security     : false\n",
      "Erase flash\n",
      "\n",
      "Done in 0.001 seconds\n",
      "Write 98972 bytes to flash (25 pages)\n",
      "\n",
      "[                              ] 0% (0/25 pages)\n",
      "[=                             ] 4% (1/25 pages)\n",
      "[==                            ] 8% (2/25 pages)\n",
      "[===                           ] 12% (3/25 pages)\n",
      "[====                          ] 16% (4/25 pages)\n",
      "[======                        ] 20% (5/25 pages)\n",
      "[=======                       ] 24% (6/25 pages)\n",
      "[========                      ] 28% (7/25 pages)\n",
      "[=========                     ] 32% (8/25 pages)\n",
      "[==========                    ] 36% (9/25 pages)\n",
      "[============                  ] 40% (10/25 pages)\n",
      "[=============                 ] 44% (11/25 pages)\n",
      "[==============                ] 48% (12/25 pages)\n",
      "[===============               ] 52% (13/25 pages)\n",
      "[================              ] 56% (14/25 pages)\n",
      "[==================            ] 60% (15/25 pages)\n",
      "[===================           ] 64% (16/25 pages)\n",
      "[====================          ] 68% (17/25 pages)\n",
      "[=====================         ] 72% (18/25 pages)\n",
      "[======================        ] 76% (19/25 pages)\n",
      "[========================      ] 80% (20/25 pages)\n",
      "[=========================     ] 84% (21/25 pages)\n",
      "[==========================    ] 88% (22/25 pages)\n",
      "[===========================   ] 92% (23/25 pages)\n",
      "[============================  ] 96% (24/25 pages)\n",
      "[==============================] 100% (25/25 pages)\n",
      "Done in 4.110 seconds\n"
     ]
    }
   ],
   "source": [
    "!arduino-cli upload -b arduino:mbed_nano:nano33ble -p COM5 kws"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20eab198",
   "metadata": {},
   "source": [
    "### Testing\n",
    "\n",
    "Below code is connecting to the Arduino board an displaying the output in a cell.\n",
    "If you say `Yes` loudly enough (so not too far from the board) and if you don't have too much background noise in your room, then it should work.\n",
    "\n",
    "You need to install the `pyserial` python package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "5a36492a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import serial\n",
    "import ipywidgets as widgets\n",
    "import time\n",
    "import threading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "7d55a8ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "STOPSERIAL=False   \n",
    "def stop_action(btn):\n",
    "    global STOPSERIAL\n",
    "    STOPSERIAL=True\n",
    "    \n",
    "out = widgets.Output(layout={'border': '1px solid black','height':'40px'})\n",
    "button = widgets.Button(\n",
    "    description='Stop',\n",
    "    disabled=False,\n",
    "    button_style='', # 'success', 'info', 'warning', 'danger' or ''\n",
    "    tooltip='Click me'\n",
    ")\n",
    "button.on_click(stop_action)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "ecabec72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d06ab7d3efa41c38f51d9e9b626b40e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Output(layout=Layout(border='1px solid black', height='40px')), Button(description='Stop', styl…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "out.clear_output()\n",
    "display(widgets.VBox([out,button]))\n",
    "\n",
    "STOPSERIAL = False\n",
    "\n",
    "def get_serial():\n",
    "    try:\n",
    "        with serial.Serial('COM6', 115200, timeout=1) as ser:  \n",
    "            ser.reset_input_buffer()\n",
    "            global STOPSERIAL\n",
    "            while not STOPSERIAL:\n",
    "                data=ser.readline()\n",
    "                if (len(data)>0):\n",
    "                    with out:\n",
    "                        out.clear_output()\n",
    "                        res=data.decode('ascii').rstrip()\n",
    "                        if res==\"Yes\":\n",
    "                            display(HTML(\"<p style='color:#00AA00';>YES</p>\"))\n",
    "                        else:\n",
    "                            print(res)\n",
    "            with out:\n",
    "                out.clear_output()\n",
    "                print(\"Communication closed\")\n",
    "    except Exception as inst:\n",
    "        with out:\n",
    "            out.clear_output()\n",
    "            print(inst)\n",
    "            \n",
    "t = threading.Thread(target=get_serial)\n",
    "t.start()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
