Opportunity
Human activity recognition using ear-worn devices (e.g., earphones) is increasingly important for health monitoring, smart interaction, and context-aware applications. However, the head movements of users introduce high dynamism and unpredictability, causing motion signals captured by inertial measurement units (IMUs) in ear-worn devices to exhibit significant variation and poor stability compared to those from traditional mobile devices like smartphones. This instability makes activity classification from ear-worn device data considerably more challenging. In prior art, achieving acceptable classification accuracy required collecting enormous amounts of labeled ear-worn device data, leading to high data acquisition and training costs. Moreover, publicly available datasets for ear-worn devices are scarce, while abundant smartphone IMU datasets exist. Therefore, there is a strong need for a method that can leverage the rich, stable data from smartphones (source domain) to assist in training a behavior recognition model for ear-worn devices (target domain), thereby drastically reducing the need for expensive target domain data collection and lowering overall training costs.
Technology
The proposed patent introduces a domain-adversarial training framework for building a behavior recognition model specifically for ear-worn devices. The method first collects first behavior data from a mobile smart terminal (e.g., smartphone) and second behavior data from an ear-worn device (e.g., in-ear headphones). Both datasets undergo preprocessing steps: low-pass filtering to remove head-motion noise (cutoff frequency of 5 Hz for ear-worn data), data augmentation via synthetic acceleration and gyroscope magnitude to enhance orientation invariance, and normalization to align value ranges between accelerometer and gyroscope readings. Training samples are constructed by selecting samples from both domains, with a ratio favoring the first (source) data by a preset threshold (e.g., over 100 times more source samples) to minimize target data usage. A feature extractor built with a bidirectional long short-term memory (Bi-LSTM) network extracts temporal features and dependencies from the training samples. The extracted features are fed into two parallel branches: a label predictor that outputs activity recognition results, and a classifier that determines whether the sample comes from the smartphone or the ear-worn device (binary domain classification). A loss function combining label prediction loss and domain classification loss is used, with a gradient reversal layer (GRL) applied to the classifier branch. During forward propagation, the GRL acts as an identity function; during backpropagation, it reverses the gradient sign, effectively maximizing the domain classification error while minimizing the label prediction error. This adversarial process forces the feature extractor to learn domain-invariant features that are discriminative for activity recognition but indistinguishable across domains. The network parameters (feature extractor, label predictor, classifier) are updated iteratively until convergence, yielding a model that can accurately recognize human activities from ear-worn device data without requiring extensive target-domain labeled data.
Advantages
- Significantly reduces data collection and training costs by leveraging abundant smartphone data to supplement limited ear-worn device data.
- Eliminates the need for large-scale labeled ear-worn device datasets, which are expensive and difficult to obtain.
- Achieves high activity recognition accuracy on ear-worn devices despite the inherent instability of head-mounted sensor signals.
- Enhances model generalization across different device domains and user orientations through domain-adversarial learning and data augmentation.
- Utilizes a gradient reversal layer to effectively learn domain-invariant features, improving robustness to head motion artifacts.
- Compatible with existing open-source smartphone IMU datasets, enabling rapid deployment and fine-tuning for ear-worn applications.
Applications
- Ear-worn devices such as wireless earbuds and smart headphones for real-time human activity recognition (walking, running, sitting, etc.).
- Wearable health monitoring systems for fitness tracking, fall detection, and rehabilitation assessment.
- Smart interaction systems where user activity context triggers device responses (e.g., adjusting music or notifications based on motion).
- Augmented reality and virtual reality headsets that need to recognize user movements for immersive experiences.
- Industrial and research applications requiring low-cost, unobtrusive human behavior analysis in daily life.
