EVENT

Event News

ASPIRE Human-Centered AI - UMCS Monthly Talk (Online)

How can AI recognize a signer's emotions when facial expressions can convey both linguistic and affective information?

Following last month's talk by Dr. Zhidong Xiao on emotion recognition and generation in multimodal interaction, this month's UMCS Monthly Talk continues our exploration of emotion and AI, with a particular focus on sign language. Prof. Kotaro Funakoshi will present recent work on emotion recognition in Japanese Sign Language, moving from isolated utterances to emotion recognition in dialogue.

He will introduce the eJSL solo and eJSL Dialog datasets and discuss what happens when existing emotion-recognition approaches are applied to signed interaction. The talk will also explore an important direction for sign language AI: moving beyond translating propositional content toward systems that can preserve a signer's affective intent in conversation.

Everyone is welcome to attend.

Title:

Recognizing Emotions in Signers

Presenter:

Prof. Kotaro Funakoshi (Institute of Integrated Research,Institute of Science Tokyo, Japan), a Co-PI of the UMCS Japanese team

Abstract:

Emotion recognition in sign language faces several fundamental challenges, including the overlap between grammatical and affective facial expressions, the scarcity of labeled training data, and the lack of datasets incorporating conversational context. To address these challenges, we investigate emotion recognition at both the isolated-utterance and dialogue levels. First, we introduce eJSL solo, a benchmark dataset in which two Japanese Sign Language signers express 78 utterances under seven emotional states, resulting in 1,092 video clips. Through cross-lingual experiments using eJSL and BOBSL, a large-scale British Sign Language dataset with subtitles, we demonstrate that spoken-language textual emotion recognition can mitigate sign-language data scarcity, that temporal segment selection substantially affects performance, and that incorporating hand motion improves emotion recognition. We then extend emotion recognition from isolated utterances to sign language conversation by introducing the eJSL Dialog dataset, which contains 1,920 video samples organized into 480 dialogues based on scripts from the STUDIES corpus. Systematic benchmarking, ranging from isolated visual models to multimodal conversational architectures, reveals a significant domain gap when generic conversational emotion recognition models are applied to sign language. Our findings highlight the need for sign-language-specific, context-aware visual representations and larger conversational datasets for large-scale pre-training. Finally, we discuss future directions toward emotion-aware sign language translation that preserves not only propositional content but also the signer's affective intent.

Bio:

Kotaro Funakoshi is currently an associate professor at the Institute of Integrated Research, Institute of Science Tokyo since 2020. Formerly with Honda Research Institute Japan Co., Ltd. from 2006, he also served as a program-specific associate professor with the Cooperative Intelligence Joint Chair at Kyoto University from 2017 to 2020. He received his B.E., M.E., and Dr.Eng. degrees from Institute of Science Tokyo (formerly Tokyo Institute of Technology) in 2000, 2002 and 2005, respectively. His research interests include natural language processing, multi-modal dialogue systems, and human-machine interaction.