Eloquent Edge ML
We finally have the Machine Learning model trained with the "Movement Sensor Dataset for Dog Behavior Classification" (Vehkaoja et al. 2022) running on our edge device: the Nicla SenseME board!

It was a rather arduous journey to get to this point! As documented in the previous labnotes, it was challenging to train a satisfactory model using Edge Impulse, and after we succedded at that we encountered an unforessen difficulty in getting the model to run in our edge device.
This last difficulty led us to explore alternative approaches, including the usage of a different sensing board with edge ML capabilities: the Seeed Studio XIAO nRF52840 Sense. In trying to make our Edge Impulse model work with the XIAO board, we discovered an intriguing library: eloquent edge-impulse. Its description "Run Edge Impulse on Arduino with ease" immediately caught our attention. Further investigation revealed that such library was part of a larger endeavor: Eloquent Arduino. Basically, an edge ML consulting company.
After a couple of emails describing our project and current bottlenecks, we quickly found an arragement to work together towards the our end goal for this phase of the project: get an edge ml model trained with "Movement Sensor Dataset for Dog Behavior Classification" (Vehkaoja et al. 2022) running on one of our devices.
Simone Salerno, the person behind Eloquent Arduino, decided that it'd actually be better to retrain the model using his expertise (and business license) with Edge Impulse, using similar parameters than the ones we had chosen but going over some of the limitations we had faced before (like data upload limits). So after some tests with the full dataset he suggested that we would combine a few classes from our original selection. So we went from ["Galloping", "Lying chest", "Sitting", "Sniffing", "Standing", "Trotting", "Walking"] to [Lying-chest, Sitting, Standing, Sniffing, Walking, Trotting]. Essetially, these groupings were made due to the innacuracy of different models to perceive the differences between those movements, as illustrated by the model we trained ourselves. This manipulation yielded good results in terms of model's confusion matrix: 93% prediction accuracy, the best we had gotten thus far.
So after that we proceeded to upload the model to the XIAO board, but the first attempt at flashing the firmware with the edge model bricked the board. This is the second time I have issues with boards Powered by Nordic nRF52840 chip in the context of this project (the other one is the board powering the vibrotactile bracelet itself), which makes me reluctant to want to engage with them in the future (Lesson #13).
Therefore, we returned to the Nicla SenseME, which presented a series of different issues. After incremental tests of the codebase, to verify that the model would upload and run, we got to a critical moment when it came to integrating the BLE functionalities to the codebase: as we knew at the onset of this project, the Nicla SenseME struggles with code that makes use of both BLE and edge-inference functionalities. Such codebases often don't have compile for this type of board, as they make the board run out of memory. At least with Arduino IDE. So one has to try to build and upload such codebases via PlatformIO, as a kind of hack to make things work (Lesson #11). Even so, when trying to do that, we were faced with another kind of issue: "fatal error: mbed_version.h: No such file or directory", which seems to have to do with Edge Impulse's SDK.
At that point, I could only rely on Simone's expertise to find a way out and thankfully he found it. By forgeting about Edge Impulse altogether, he "used a custom implementation inspired by the Python package scikit-learn, made of:
- feature scaling (move all the data into the 0-1 range): https://scikit-learn.org/1.6/modules/generated/sklearn.preprocessing.MinMaxScaler.html
- a "manual" windowing approach, segmenting the data into chunks of 2 seconds
- for each window, extracted statistical moments to be used as features (min, max, mean, std, skew, kurtosis)
- classification using Random Forest (https://scikit-learn.org/1.6/modules/generated/sklearn.ensemble.RandomForestClassifier.html) "

In essence, a similar model to the Edge Impulse one, but with more freedom for deployment. And despite the lowering of overall accuracy, wow, it worked! We could then upload the code to the Nicla SenseME, working with the edge-inference and the BLE functionalities! Well, kind of... Even though the code uploaded and ran with the predictions being printed over serial, I have not been able to communicate with the board over BLE yet, so as to make use of the predictions in whichever way I want. This is the next step in this journey! In the meantime, another lesson that became clear (lesson #15) after all of this is that:
A dataset collected using specific hardware is unlikely to produce a high-performing model when applied to different hardware, despite overall accuracy during training. This is something that we are observing in practice. The dataset we are using has not been collected with a Nicla SenseME. Thus, prediction results in the Nicla board are not matching actual movements very well. There's a tendency for movements to be categorized as either Sniffing or Trotting, while walking rarely gets predicted. At least while I test the system moving it with my hands. Let's wait to see if things are different when it gets worn by Maniçoba.
Up next
As mentioned, the next immediate step in the process is accessing both the current predictions and raw accelerometer data via BLE, so we can play with them in the haptics interface. Once those aspects are addressed, the project formally comes to an end (in the context of this experiment grant), but as I have mentioned in a previous labnote:
"another idea that has been growing within us is the use of a heart rate sensor alongside the accelerometer predictions, as a way to couple movement and physiological data, all at while transfering both data inputs to the bracelet, so we can really feel our dog's heartbets."
And with the learned lesson around data gathering accross different pieces of hardware, there's also an emerging wish to start collecting data for our own investigations, potentially making use of the current ML predictions (despite their actual accuracy - at least as proxy) and physiological data. But let's see. First, let's finish what we said we'd do!
2 comments