Dataset Information

Perils and opportunities in using large language models in psychological research.

ABSTRACT: The emergence of large language models (LLMs) has sparked considerable interest in their potential application in psychological research, mainly as a model of the human psyche or as a general text-analysis tool. However, the trend of using LLMs without sufficient attention to their limitations and risks, which we rhetorically refer to as "GPTology", can be detrimental given the easy access to models such as ChatGPT. Beyond existing general guidelines, we investigate the current limitations, ethical implications, and potential of LLMs specifically for psychological research, and show their concrete impact in various empirical studies. Our results highlight the importance of recognizing global psychological diversity, cautioning against treating LLMs (especially in zero-shot settings) as universal solutions for text analysis, and developing transparent, open methods to address LLMs' opaque nature for reliable, reproducible, and robust inference from AI-generated data. Acknowledging LLMs' utility for task automation, such as text annotation, or to expand our understanding of human psychology, we argue for diversifying human samples and expanding psychology's methodological toolbox to promote an inclusive, generalizable science, countering homogenization, and over-reliance on LLMs.

SUBMITTER: Abdurahman S

PROVIDER: S-EPMC11249969 | biostudies-literature | 2024 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Perils and opportunities in using large language models in psychological research.

Abdurahman Suhaib S Atari Mohammad M Karimi-Malekabadi Farzan F Xue Mona J MJ Trager Jackson J Park Peter S PS Golazizian Preni P Omrani Ali A Dehghani Morteza M

PNAS nexus 20240716 7

The emergence of large language models (LLMs) has sparked considerable interest in their potential application in psychological research, mainly as a model of the human psyche or as a general text-analysis tool. However, the trend of using LLMs without sufficient attention to their limitations and risks, which we rhetorically refer to as "GPTology", can be detrimental given the easy access to models such as ChatGPT. Beyond existing general guidelines, we investigate the current limitations, ethi ...[more]

PMID: 39015547

Similar Datasets

Project description:ImportanceOver the past decade, the leadership of the National Institute of Mental Health (NIMH) has emphasized the importance of a transdiagnostic approach to psychiatric investigation using Research Domain Criteria (RDoC) mapping more closely to neurobiology.ObjectiveTo investigate whether research support from the NIMH for individual RDoC domains and for transdiagnostic investigation has changed over time and has had differential impact in terms of publication, citations, or patent filings.Design, setting, and participantsIn this longitudinal cohort study, all R01, R21, and R03 studies funded by the NIMH between January 2003 and December 2023 were identified via the National Institutes of Health RePORTER database. Their abstracts were characterized in terms of RDoC domains (negative valence, positive valence, cognition, social, arousal, and sensorimotor) using a large language model.Main outcomes and measuresPrimary outcomes were publications, citation impact estimated at 5 years from the index year of funding, and patents, examined using regression models adjusted for other grant characteristics.ResultsAmong 8897 R01, R03, and R21 projects, reflecting $17.7 billion of investment, abstracts of 3141 (35.3%) reflected negative valence; 1344 (15.1%), positive valence; 2781 (31.3%), cognition; 1607 (18.1%), social; 343 (3.9%), arousal; and 571 (6.4%), sensorimotor domains. A total of 1793 (20.2%) incorporated a transdiagnostic perspective. Positive and social domains were associated with fewer publications (difference, -1.13 [95% CI, -2.11 to -0.15] and -2.23 [95% CI, -3.15 to -1.30], respectively) and lesser citation impact (difference, -0.47 [95% CI, -0.75 to -0.18] and -1.19 [95% CI, -1.46 to -0.91], respectively) at 5 years. Social (adjusted odds ratio [AOR], 0.11; 95% CI, 0.04-0.23) and cognitive (AOR, 0.66; 95% CI, 0.48-0.89) domains and transdiagnostic proposals (AOR, 0.37; 95% CI, 0.21-0.60) were associated with lower likelihood of patent filing.Conclusions and relevanceIn this study of NIMH funding, grants reflecting different RDoC domains differed substantially in their scientific impact in terms of publications, citations, and patent generation. The findings suggest that large language models represent a promising approach to characterizing research proposals at scale, which may be useful in guiding resource allocation to maximize scientific return on investment.

Dataset Information

Perils and opportunities in using large language models in psychological research.

Publications

Perils and opportunities in using large language models in psychological research.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets