対話者の非言語行動のマルチモーダル相乗作用解明のための機能スペクトラム解析
项目来源
项目主持人
项目受资助机构
项目编号
立项年度
立项时间
研究期限
项目级别
受资助金额
学科
学科代码
基金类别
关键词
参与者
参与机构
1.Exploring Multimodal Nonverbal Functional Features for Predicting the Subjective Impressions of Interlocutors
- 关键词：
- Facial expression; feature selection; group meeting; head movement;multimodal recognition; nonverbal communication; social signal;subjective impression; group meeting; head movement; multimodalrecognition; nonverbal communication; social signal; subjectiveimpression;FEATURE-SELECTION; CONVERSATION; PERSONALITY; JAPANESE; GAZE
- Ito, Koya;Ishii, Yoko;Ishii, Ryo;Eitoku, Shin-Ichiro;Otsuka, Kazuhiro
- 《IEEE ACCESS》
- 2024年
- 12卷
- 期
- 期刊
This paper proposes models for predicting the subjective impressions of interlocutors in discussions according to multimodal nonverbal behaviors. To that end, we focus mainly on the functional aspects of head movement and facial expressions as insightful cues. For example, head movement functions include the speaker's rhythm and the listener's back channel and thinking processes, as well as their positive emotions. Facial expression functions include emotional expressions and communicative functions such as the speaker addressing the listener and the listener's affirmation. In addition, our model employs synergetic functions, which are jointly performed with head movements and facial expressions, assuming that the simultaneous appearance of head and face functions could strengthen the results or lead to multiplexing. On the basis of these nonverbal functions, we define a set of functional features, including the rate of occurrence and composition balance among different functions that emerge during conversation. Then, a feature selection scheme is used to identify the best combinations of intermodal and intramodal features. In the experiments, an SA-Off corpus of 17 groups of discussions involving 4 female participants was used, including interlocutors' self-reported scores for 16 impression items felt during the discussion, such as helpfulness and interest. The experiments confirmed that our models' predictions were significantly correlated with the self-reported scores for more than 70% of the impression items. These results indicate the effectiveness of multimodal nonverbal functional features for predicting subjective impressions.
...2.Synergistic Functional Spectrum Analysis: A Framework for Exploring the Multifunctional Interplay Among Multimodal Nonverbal Behaviours in Conversations
- 关键词：
- Convolution;Matrix algebra;Non-negative matrix factorization;Regression analysis;Spectrum analyzers;Vector spaces;Conversation;Convolutional neural network;Functionals;Multi-modal;Multifunctionals;Multimodal nonverbal behavior;Non-verbal behaviours;Nonnegative matrix factorization;Spectra analysis;Spectra's
- Imamura, Mai;Tashiro, Ayane;Kumano, Shiro;Otsuka, Kazuhiro
- 《IEEE Transactions on Affective Computing》
- 2024年
- 卷
- 期
- 期刊
A novel framework named the synergistic functional spectrum analysis (sFSA) is proposed to explore the multifunctional interplay among multimodal nonverbal behaviours in human conversations. This study aims to reveal how multimodal nonverbal behaviours cooperatively perform communicative functions in conversations. To capture the intrinsic nature of nonverbal expressions, functional multiplicity, and interpretational ambiguity, e.g., a single head nod could imply listening, agreeing, or both, a novel concept named the functional spectrum, which is defined as the distribution of perceptual intensities of multiple functions by multiple observers, is introduced in the sFSA. Based on this concept, this paper presents functional spectrum corpora, which target 44 facial expression and 32 head movement functions. Then, spectrum decomposition is conducted to reduce the multimodal functional spectrum to a synergetic functional spectrum in a lower dimension functional space that is spanned by functional basis vectors representing primary and distinctive functionalities across multiple modalities. To that end, we propose a semiorthogonal nonnegative matrix factorization (SO-NMF) method, which assumes the additivity of multiple functions and aims to balance the distinctiveness and expressiveness of the factorization. The results confirm that some primary functional bases can be identified, which can be interpreted as the listener’s backchannel, thinking, and affirmative response functions, and the speaker’s thinking and addressing functions, and their positive emotion functions. In addition, regression models based on convolutional neural networks (CNNs) are presented to estimate the synergistic functional spectrum from the head poses and facial action units measured from conversation data. The results of these analyses and experiments confirm the potential of the sFSA and may lead to future extensions. © 2010-2012 IEEE.
...