Dataset Information

Speech Driven Gaze in a Face-to-Face Interaction.

ABSTRACT: Gaze and language are major pillars in multimodal communication. Gaze is a non-verbal mechanism that conveys crucial social signals in face-to-face conversation. However, compared to language, gaze has been less studied as a communication modality. The purpose of the present study is 2-fold: (i) to investigate gaze direction (i.e., aversion and face gaze) and its relation to speech in a face-to-face interaction; and (ii) to propose a computational model for multimodal communication, which predicts gaze direction using high-level speech features. Twenty-eight pairs of participants participated in data collection. The experimental setting was a mock job interview. The eye movements were recorded for both participants. The speech data were annotated by ISO 24617-2 Standard for Dialogue Act Annotation, as well as manual tags based on previous social gaze studies. A comparative analysis was conducted by Convolutional Neural Network (CNN) models that employed specific architectures, namely, VGGNet and ResNet. The results showed that the frequency and the duration of gaze differ significantly depending on the role of participant. Moreover, the ResNet models achieve higher than 70% accuracy in predicting gaze direction.

SUBMITTER: Arslan Aydin U

PROVIDER: S-EPMC7970197 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Speech Driven Gaze in a Face-to-Face Interaction.

Arslan Aydin Ülkü Ü Kalkan Sinan S Acartürk Cengiz C

Frontiers in neurorobotics 20210304

Gaze and language are major pillars in multimodal communication. Gaze is a non-verbal mechanism that conveys crucial social signals in face-to-face conversation. However, compared to language, gaze has been less studied as a communication modality. The purpose of the present study is 2-fold: (i) to investigate gaze direction (i.e., aversion and face gaze) and its relation to speech in a face-to-face interaction; and (ii) to propose a computational model for multimodal communication, which predic ...[more]

PMID: 33746729

Similar Datasets

Project description:BackgroundPrevious eye-tracking research on autistic individuals has mostly examined the gaze behavior of one individual in response to social stimuli presented on a computer screen, suggesting that there is atypical gaze behavior. However, it is unknown how these findings translate to the interactive dynamics of gaze behavior during "face-to-face" encounters between two individuals. Only by analyzing the gaze behaviour of both interaction partners is it possible to determine the frequency of actual eye-contact and who initiates or breaks such periods of mutual eye gaze. The knowledge gained from this analysis could contribute to theorizing about the psychological mechanisms (e.g., gaze avoidance vs. gaze indifference) underlying autism.MethodsThe present study applied a novel dual eye-tracking setup that allows the assessment and analysis of the interactive dynamics of gaze behavior regarding (i) mutual eye gaze (i.e., eye contact), (ii) initiations, and (iii) break-ups of eye contact. Participants (37 autistic individuals, 37 age- and IQ-matched neurotypical individuals) performed a semi-standardized social interaction (i.e., Fast Friends Procedure) with a confederate (trained to interact in a standardized manner).ResultsEye contact was reduced in interactions involving autistic individuals. Additional analyses revealed that this reduction was primarily due to the more frequent breaking of eye contact by these individuals. We also found considerable heterogeneity among autistic individuals, with atypical gaze behavior present in only about half of the sample.LimitationsFurther research is required to determine whether the interactive dynamics of gaze behavior observed in this dual eye-tracking setup can be generalized to real-world situations. Future studies could also include arousal-related physiological measures.ConclusionsBy tracking the gaze behavior of two interacting individuals, this study reveals specific atypicalities in the interactive dynamics of gaze behavior in a subset of autistic individuals, potentially informing diagnostic and therapeutic decisions. More broadly, our study highlights the added value of dual eye-tracking in elucidating the interactive nature of social encounters in both neurodiverse and neurotypical individuals.Trial registrationThe study was registered as a clinical trial before starting data collection ( https://drks.de/search/en/trial/DRKS00018957 ; Registration Date: 12/17/2019).

Dataset Information

Speech Driven Gaze in a Face-to-Face Interaction.

Publications

Speech Driven Gaze in a Face-to-Face Interaction.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets