Prompt architecture induces methodological artifacts in large language models.

ABSTRACT: We examine how the seemingly arbitrary way a prompt is posed, which we term "prompt architecture," influences responses provided by large language models (LLMs). Five large-scale, full-factorial experiments performing standard (zero-shot) similarity evaluation tasks using GPT-3, GPT-4, and Llama 3.1 document how several features of prompt architecture (order, label, framing, and justification) interact to produce methodological artifacts, a form of statistical bias. We find robust evidence that these four elements unduly affect responses across all models, and although we observe differences between GPT-3 and GPT-4, the changes are not necessarily for the better. Specifically, LLMs demonstrate both response-order bias and label bias, and framing and justification moderate these biases. We

SUBMITTER: Brucks M 

PROVIDER: S-EPMC12036937 | biostudies-literature | 2025

REPOSITORIES: biostudies-literature