Human-anchored longitudinal comparison of generative AI with a bias-calibrated LLM-as-judge.

ABSTRACT: Service LLMs evolve without public changelogs, complicating reproducible evaluation. We present a preregistered human-anchored longitudinal study that tracks three major model families over ten weekly waves using a fixed prompt bank (N = 240) across six domains. Blinded human raters provided correctness judgments, and a bias-calibrated LLM-as-judge produced secondary pairwise preferences corrected weekly via a Bradley-Terry model. Mixed-effects modeling and change-point detection (PELT with MBIC penalty) identified significant service drift patterns. Results show divergent stability trajectories among models: one stable, one improving, and one degrading mid-study. Judge calibration increased agreement with humans (τ = 0.59-0.68) while reducing volatility. Safety metrics co-varied with drif

SUBMITTER: Wiese T 

PROVIDER: S-EPMC12863567 | biostudies-literature | 2026

REPOSITORIES: biostudies-literature