A Unified Uncertainty-Aware and Imbalance-Resilient Framework for Robust Multimodal Emotion Recognition

Abstract

The multimodal emotion recognition has become a key field of research because of its importance in human–computer interaction, affective computing and intelligent decision support systems. But these techniques are generally hampered by modality imbalance, class distribution imbalance and propagation of uncertainty across heterogeneous feature space. To solve these problems, the proposed study provides a unified and modular multimodal learning framework based on four complementary enhancement mechanisms to boost the robustness, generalization, and interpretability. The proposed framework consists of (i) an uncertainty-aware adaptive gating mechanism to control the contributions from the different modality, (ii) a multi-sample dropout strategy for better regularization and to prevent overfitting, (iii) a contrastive representation alignment module to ensure the consistency of the different modalities, and (iv) a logit-adjusted learning scheme to deal with the issue of class imbalance. All are organized as separate components, which allows for flexible activation and for systematic ablation analysis. The proposed framework consistently outperforms baseline emotion models and all individual variants of the proposed enhancement on the weighted F1-score, macro F1-score, and overall classification accuracy on extensive experiments conducted on a large-scale multimodal emotion dataset. The ablation results also validate the contribution of uncertainty modeling and imbalance-aware learning, as they yield the best performance. Interestingly, the full configuration works well on CPU-based environments and provides the best generalization performance. The work presented in this thesis makes a significant contribution to the scalable, interpretable, and practically deployable multimodal emotion recognition framework, and provides insights into future research for developing adaptive multimodal learning and powerful affective computing systems.