Performance Evaluation of 18 Generative AI Models (ChatGPT, Gemini, Claude, and Perplexity) in 2024 Japanese Pharmacist Licensing Examination: Comparative Study.

ABSTRACT:

Background

Generative artificial intelligence (AI) has shown rapid advancements and increasing applications in various domains, including health care. Previous studies have evaluated AI performance on medical license examinations, primarily focusing on ChatGPT. However, the availability of new online chat-based large language models (OC-LLMs) and their potential utility in pharmacy licensing examinations remain underexplored. Considering that pharmacists require a broad range of expertise in physics, chemistry, biology, and pharmacology, verifying the knowledge base and problem-solving abilities of these new models in Japanese pharmacy examinations is necessary.

Objective

This study aimed to assess the performance of 18 OC-LLMs released in 2024 in the 107th Japanese National

SUBMITTER: Sato H 

PROVIDER: S-EPMC12445623 | biostudies-literature | 2025 Sep

REPOSITORIES: biostudies-literature