GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet.

ABSTRACT:

Background

Large language models (LLMs) that can efficiently screen and identify studies meeting specific criteria would streamline literature reviews. Additionally, those capable of extracting data from publications would enhance knowledge discovery by reducing the burden on human reviewers.

Methods

We created an automated pipeline utilizing OpenAI GPT-4 32 K API version "2023-05-15" to evaluate the accuracy of the LLM GPT-4 responses to queries about published papers on HIV drug resistance (HIVDR) with and without an instruction sheet. The instruction sheet contained specialized knowledge designed to assist a person trying to answer questions about an HIVDR paper. We designed 60 questions pertaining to HIVDR and created markdown versions of 60 published HIVDR papers in Pub

SUBMITTER: Tao K 

PROVIDER: S-EPMC11197181 | biostudies-literature | 2024 Jun

REPOSITORIES: biostudies-literature