LLM-as-a-Judge: automated evaluation of search query parsing using large language models.

ABSTRACT:

Introduction

The adoption of Large Language Models (LLMs) in search systems necessitates new evaluation methodologies beyond traditional rule-based or manual approaches.

Methods

We propose a general framework for evaluating structured outputs using LLMs, focusing on search query parsing within an online classified platform. Our approach leverages LLMs' contextual reasoning capabilities through three evaluation methodologies: Pointwise, Pairwise, and Pass/Fail assessments. Additionally, we introduce a Contextual Evaluation Prompt Routing strategy to improve reliability and reduce hallucinations.

Results

Experiments conducted on both small- and large-scale datasets demonstrate that LLM-based evaluation achieves approximately 90% agreement with human judgments.

Discus

SUBMITTER: Baysan MS 

PROVIDER: S-EPMC12319771 | biostudies-literature | 2025

REPOSITORIES: biostudies-literature