Kavli Affiliate: Zhuo Li
| First 5 Authors: Jinpeng Hu, Jinpeng Hu, , ,
| Summary:
Multimodal large language models (MLLMs) have been widely applied across
various fields due to their powerful perceptual and reasoning capabilities. In
the realm of psychology, these models hold promise for a deeper understanding
of human emotions and behaviors. However, recent research primarily focuses on
enhancing their emotion recognition abilities, leaving the substantial
potential in emotion reasoning, which is crucial for improving the naturalness
and effectiveness of human-machine interactions. Therefore, in this paper, we
introduce a multi-turn multimodal emotion understanding and reasoning (MTMEUR)
benchmark, which encompasses 1,451 video data from real-life scenarios, along
with 5,101 progressive questions. These questions cover various aspects,
including emotion recognition, potential causes of emotions, future action
prediction, etc. Besides, we propose a multi-agent framework, where each agent
specializes in a specific aspect, such as background context, character
dynamics, and event details, to improve the system’s reasoning capabilities.
Furthermore, we conduct experiments with existing MLLMs and our agent-based
method on the proposed benchmark, revealing that most models face significant
challenges with this task.
| Search Query: ArXiv Query: search_query=au:”Zhuo Li”&id_list=&start=0&max_results=3