Danilo

Danilo

Jan 24, 2025

Group 6 Copy 1,013
2

Eloquent Edge ML

We finally have the Machine Learning model trained with the "Movement Sensor Dataset for Dog Behavior Classification" (Vehkaoja et al. 2022) running on our edge device: the Nicla SenseME board!

Nicla SenseME running edge ML model

It was a rather arduous journey to get to this point! As documented in the previous labnotes, it was challenging to train a satisfactory model using Edge Impulse, and after we succedded at that we encountered an unforessen difficulty in getting the model to run in our edge device.

This last difficulty led us to explore alternative approaches, including the usage of a different sensing board with edge ML capabilities: the Seeed Studio XIAO nRF52840 Sense. In trying to make our Edge Impulse model work with the XIAO board, we discovered an intriguing library: eloquent edge-impulse. Its description "Run Edge Impulse on Arduino with ease" immediately caught our attention. Further investigation revealed that such library was part of a larger endeavor: Eloquent Arduino. Basically, an edge ML consulting company.

After a couple of emails describing our project and current bottlenecks, we quickly found an arragement to work together towards the our end goal for this phase of the project: get an edge ml model trained with "Movement Sensor Dataset for Dog Behavior Classification" (Vehkaoja et al. 2022) running on one of our devices.

Simone Salerno, the person behind Eloquent Arduino, decided that it'd actually be better to retrain the model using his expertise (and business license) with Edge Impulse, using similar parameters than the ones we had chosen but going over some of the limitations we had faced before (like data upload limits). So after some tests with the full dataset he suggested that we would combine a few classes from our original selection. So we went from ["Galloping", "Lying chest", "Sitting", "Sniffing", "Standing", "Trotting", "Walking"] to [Lying-chest, Sitting, Standing, Sniffing, Walking, Trotting]. Essetially, these groupings were made due to the innacuracy of different models to perceive the differences between those movements, as illustrated by the model we trained ourselves. This manipulation yielded good results in terms of model's confusion matrix: 93% prediction accuracy, the best we had gotten thus far.

So after that we proceeded to upload the model to the XIAO board, but the first attempt at flashing the firmware with the edge model bricked the board. This is the second time I have issues with boards Powered by Nordic nRF52840 chip in the context of this project (the other one is the board powering the vibrotactile bracelet itself), which makes me reluctant to want to engage with them in the future (Lesson #13).

Therefore, we returned to the Nicla SenseME, which presented a series of different issues. After incremental tests of the codebase, to verify that the model would upload and run, we got to a critical moment when it came to integrating the BLE functionalities to the codebase: as we knew at the onset of this project, the Nicla SenseME struggles with code that makes use of both BLE and edge-inference functionalities. Such codebases often don't have compile for this type of board, as they make the board run out of memory. At least with Arduino IDE. So one has to try to build and upload such codebases via PlatformIO, as a kind of hack to make things work (Lesson #11). Even so, when trying to do that, we were faced with another kind of issue: "fatal error: mbed_version.h: No such file or directory", which seems to have to do with Edge Impulse's SDK.

At that point, I could only rely on Simone's expertise to find a way out and thankfully he found it. By forgeting about Edge Impulse altogether, he "used a custom implementation inspired by the Python package scikit-learn, made of:

 - feature scaling (move all the data into the 0-1 range): https://scikit-learn.org/1.6/modules/generated/sklearn.preprocessing.MinMaxScaler.html

- a "manual" windowing approach, segmenting the data into chunks of 2 seconds

- for each window, extracted statistical moments to be used as features (min, max, mean, std, skew, kurtosis)

- classification using Random Forest (https://scikit-learn.org/1.6/modules/generated/sklearn.ensemble.RandomForestClassifier.html) "

New model's confusion matrix

In essence, a similar model to the Edge Impulse one, but with more freedom for deployment. And despite the lowering of overall accuracy, wow, it worked! We could then upload the code to the Nicla SenseME, working with the edge-inference and the BLE functionalities! Well, kind of... Even though the code uploaded and ran with the predictions being printed over serial, I have not been able to communicate with the board over BLE yet, so as to make use of the predictions in whichever way I want. This is the next step in this journey! In the meantime, another lesson that became clear (lesson #15) after all of this is that:

  • A dataset collected using specific hardware is unlikely to produce a high-performing model when applied to different hardware, despite overall accuracy during training. This is something that we are observing in practice. The dataset we are using has not been collected with a Nicla SenseME. Thus, prediction results in the Nicla board are not matching actual movements very well. There's a tendency for movements to be categorized as either Sniffing or Trotting, while walking rarely gets predicted. At least while I test the system moving it with my hands. Let's wait to see if things are different when it gets worn by Maniçoba.


Up next

As mentioned, the next immediate step in the process is accessing both the current predictions and raw accelerometer data via BLE, so we can play with them in the haptics interface. Once those aspects are addressed, the project formally comes to an end (in the context of this experiment grant), but as I have mentioned in a previous labnote:

"another idea that has been growing within us is the use of a heart rate sensor alongside the accelerometer predictions, as a way to couple movement and physiological data, all at while transfering both data inputs to the bracelet, so we can really feel our dog's heartbets."

And with the learned lesson around data gathering accross different pieces of hardware, there's also an emerging wish to start collecting data for our own investigations, potentially making use of the current ML predictions (despite their actual accuracy - at least as proxy) and physiological data. But let's see. First, let's finish what we said we'd do!

References
  • 1. Vehkaoja, A., Somppi, S., Törnqvist, H., Valldeoriola Cardó, A., Kumpulainen, P., Väätäjä, H., Majaranta, P., Surakka, V., Kujala, M. V., & Vainio, O. (2022). Description of movement sensor dataset for dog behavior classification. Data in Brief, 40, 107822. https://doi.org/10.1016/j.dib.2022.107822
Tags

2 comments

Join the conversation!Sign In
  • tom
    tom
    Looking for high-quality metal detectors and professional treasure hunting equipment? Visit Detectoryab for the latest models, expert reviews, and reliable devices to enhance your search. Start your journey today https://detectoryab.com/
    Feb 18, 2025
  • johan
    johan
    You've shared some really valuable insights into the challenges and solutions in your edge ML project. The workaround with scikit-learn is particularly impressive. Discover what the app offers could offer tools to further simplify your process. Looking forward to your next updates!
    Jan 29, 2025

About This Project

Edge machine learning refers to the process of running embedded ML models on site using devices capable of collecting, processing, and recognizing patterns within collections of raw data. This project seeks to train one of such devices (Nicla SenseME) with dog data from the Earth Species' Bio-logger Ethogram Benchmark (BEBE). The board will be used as a smart dog collar, with its ML inferences controlling haptics vibrations in a bracelet wore by a person.

Blast off!

Browse Other Projects on Experiment

Related Projects

Why are jokes funny?

The first step of my research is to come up with theories for why jokes are funny. Next we will test those...

Generative Design of Programmable Metal-Binding Proteins for Bioremediation

The mining of heavy metals accounts for about 10% of global greenhouse gas emissions. Moreover, exposure...

Designing ultrastable carbonic anhydrase with deep generative models and high-throughput assays

To minimize the impact of CO2 emissions on life on earth, we need technologies for carbon capture exceeding...

Backer Badge Funded

A computer science project funded by 5 people

Add a comment