Dataset Information

Implicit Perception of Differences between NLP-Produced and Human-Produced Language in the Mentalizing Network.

ABSTRACT: Natural language processing (NLP) is central to the communication with machines and among ourselves, and NLP research field has long sought to produce human-quality language. Identification of informative criteria for measuring NLP-produced language quality will support development of ever-better NLP tools. The authors hypothesize that mentalizing network neural activity may be used to distinguish NLP-produced language from human-produced language, even for cases where human judges cannot subjectively distinguish the language source. Using the social chatbots Google Meena in English and Microsoft XiaoIce in Chinese to generate NLP-produced language, behavioral tests which reveal that variance of personality perceived from chatbot chats is larger than for human chats are conducted, suggesting that chatbot language usage patterns are not stable. Using an identity rating task with functional magnetic resonance imaging, neuroimaging analyses which reveal distinct patterns of brain activity in the mentalizing network including the DMPFC and rTPJ in response to chatbot versus human chats that cannot be distinguished subjectively are conducted. This study illustrates a promising empirical basis for measuring the quality of NLP-produced language: adding a judge's implicit perception as an additional criterion.

SUBMITTER: Wei Z

PROVIDER: S-EPMC10131862 | biostudies-literature | 2023 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Implicit Perception of Differences between NLP-Produced and Human-Produced Language in the Mentalizing Network.

Wei Zhengde Z Chen Ying Y Zhao Qian Q Zhang Pengyu P Zhou Longxi L Ren Jiecheng J Piao Yi Y Qiu Bensheng B Xie Xing X Wang Suiping S Liu Jia J Zhang Daren D Kadosh Roi Cohen RC Zhang Xiaochu X

Advanced science (Weinheim, Baden-Wurttemberg, Germany) 20230207 12

Natural language processing (NLP) is central to the communication with machines and among ourselves, and NLP research field has long sought to produce human-quality language. Identification of informative criteria for measuring NLP-produced language quality will support development of ever-better NLP tools. The authors hypothesize that mentalizing network neural activity may be used to distinguish NLP-produced language from human-produced language, even for cases where human judges cannot subjec ...[more]

PMID: 36748300

Similar Datasets

Project description:The concept of mentalizing has been widely studied, but almost exclusively through tasks with explicit instructions. Recent studies suggest that people also mentalize on a more implicit level. However, to our knowledge, no study to date has directly contrasted the effects of implicit and explicit mentalizing processes on an implicit dependent measure within-subjects. We implemented this by using two versions of an object detection task, differing only on secondary catch questions. We hypothesized that if explicit mentalizing relies on complementary processes beyond those underlying implicit mentalizing, this would be reflected in enhanced belief effects in the explicit version. Twenty-eight healthy adults watched movies in which, during the first phase, both they themselves and another agent formed a belief about the location of a ball, and although irrelevant, these beliefs could influence their ball detection reaction times in the second phase. After this response phase, there were occasional catch questions that were different for the explicit and implicit task version. Finally, self-report measures of autism spectrum disorder (ASD) symptomatology were included, as the literature suggests that ASD is related to a specific deficit in implicit mentalizing. Both in the explicit and implicit version, belief conditions had a significant effect on reaction times, with responses being slower when neither the participant nor the other agent expected the ball to be present compared to all other conditions. Importantly, after the implicit version, participants reported no explicit mentalizing awareness. In our neurotypical sample, ASD symptoms were not found to correlate with either explicit or implicit mentalizing. In conclusion, the reaction time patterns in the explicit and implicit version of the task show strikingly similar effects of mentalizing, indicating that participants processed beliefs to the same extent regardless of whether they mentalized explicitly or implicitly, with no additional effects for explicit processing.

Dataset Information

Implicit Perception of Differences between NLP-Produced and Human-Produced Language in the Mentalizing Network.

Publications

Implicit Perception of Differences between NLP-Produced and Human-Produced Language in the Mentalizing Network.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets