Human Expertise and Large Language Model Embeddings in the Content Validity Assessment of Personality Tests.

ABSTRACT: In this article, we explore the application of Large Language Models (LLMs) in assessing the content validity of psychometric instruments, focusing on the Big Five Questionnaire (BFQ) and Big Five Inventory (BFI). Content validity, a cornerstone of test construction, ensures that psychological measures adequately cover their intended constructs. Using both human expert evaluations and advanced LLMs, we compared the accuracy of semantic item-construct alignment. Graduate psychology students employed the Content Validity Ratio to rate test items, forming the human baseline. In parallel, state-of-the-art LLMs, including multilingual and fine-tuned models, analyzed item embeddings to predict construct mappings. The results reveal distinct strengths and limitations of human and AI approaches. H

SUBMITTER: Milano N 

PROVIDER: S-EPMC12356817 | biostudies-literature | 2025 Aug

REPOSITORIES: biostudies-literature