Large language model performance versus human expert ratings in automated suicide risk assessment.

ABSTRACT: Large Language Models' (LLMs) potential for psychological diagnostics requires systematic evaluation. We aimed to investigate conditions for reliable and valid psychological assessments, focusing on suicide risk evaluation in clinical data by comparing LLM-generated ratings with human expert ratings across model configurations., analyzing 100 youth crisis text line conversation transcripts rated by four experts using the Nurses' Global Assessment of Suicide Scale (NGASR). Using Mixtral-8x7B-Instruct, we generated ratings across three temperature settings and prompting styles (zero-shot, few-shot, chain-of-thought). Across configurations we compared (a) inter-rating-reliability for AI-generated NGASR risk and sum scores, (b) LLM-to-human observer agreement regarding sum score, risk category

SUBMITTER: Thomas J 

PROVIDER: S-EPMC12603332 | biostudies-literature | 2025 Nov

REPOSITORIES: biostudies-literature