EPV641
When the Body Speaks and the Tone Echoes: Exploring GAI Multimodal Emotion Recognition
H. Dery 1,*, D. Piterman 1, Z. Elyoseph 1 2, E. Refoua 3, G. Meinlschmidt 4 5, K. Bar 6, D. Hadar Shoval 7, A. Geller 8
1School of Counseling and Human Development, University of Haifa, Haifa, Israel, 2Department of Brain Science, Imperial College, London, United Kingdom, 3Department of Psychology, Bar-Ilan University, Ramat-Gan, Israel, 4Department of Psychology, Trier University, Trier, Germany, 5Department of Digital and Blended Psychosomatics and Psychotherapy, University of Basel and University Hospital Basel, Basel, Switzerland, 6School of Computer Science, Reichman University, Herzliya, 7Department of Psychology and Educational Counseling, Max Stern Yezreel Valley College, Emek Yezreel, 8Ruppin Academic Center, Emek Hefer, Israel
Introduction: This study examined the social-cognitive capabilities of three Generative Artificial Intelligence (GAI) models from the Gemini family—Gemini Pro 1.5, Gemini Pro 2, and Gemini Flash 2—focusing on their ability to recognize human emotions from nonverbal cues, specifically vocal tone and bodily gestures. Grounded in mentalization theory—the capacity to attribute mental states to oneself and others, which underpins adaptive social functioning and therapeutic change—the research investigated the extent to which GAI models can approximate human-like emotion recognition.
Objectives: The research investigated the extent to which GAI models can approximate human-like emotion recognition from nonverbal channels and whether the accuracy of this recognition is influenced by emotional valence (positive vs. negative).
Methods: The study employed a comparative experimental design. Stimuli were drawn from the validated EU-Emotion Stimulus Set (O'Reilly et al., 2016) and EU-Emotion Voice Database (Lassalle et al., 2019). The dataset included 36 dynamic video clips representing bodily gestures and 42 audio recordings conveying vocal expressions. The GAI models were evaluated using zero-shot prompting under standardized conditions. Altogether, the study comprised 9,360 independent trials across both modalities. Human accuracy benchmarks were derived from the normative validation data of the EU-Emotion datasets, allowing for direct and controlled comparisons between model and human performance.
Results: The findings revealed both modality- and valence-dependent patterns. GAI models demonstrated performance comparable to human participants in vocal emotion recognition; Gemini Pro 2 achieved an accuracy rate of 47.14%, closely matching the human benchmark of 45.19%. In contrast, the models underperformed in the bodily gesture modality, where human participants achieved 77.08% accuracy, whereas Gemini Pro 2 reached 71.39%. A consistent valence-based performance bias was observed: all GAI models demonstrated significantly higher accuracy for positive emotions than for negative ones, particularly in the bodily modality. In contrast, human participants showed no significant difference in recognition accuracy for positive and negative emotions. All models consistently performed better with bodily gestures than with vocal expressions.
Image 1:
Image 2:
Image 3:
Conclusions: This study provides detailed empirical evidence of GAI’s present capacity for nonverbal emotion recognition. While some models—particularly Gemini Pro 2—approach human performance in specific contexts, notable gaps remain, especially in recognizing negative affect conveyed through bodily gestures. These findings underscore the promise of GAI in socially embedded applications such as clinical support, while highlighting the need for continued development using ecologically valid, diverse, and multimodal datasets to enhance the emotional competence of artificial agents.
Disclosure of Interest: None Declared