Evaluating the Performance of Artificial Intelligence Chatbots in Medical Examinations: A Scoping Review

Background and Aims: Advancements in large language models (LLMs), such as ChatGPT, have opened new avenues for medical education. Medical examinations, as standard measures of physicians’ knowledge and competence, provide an ideal context for evaluating the capabilities of these chatbots. This study aims to investigate the performance of AI chatbots in medical examinations. Methods: This scoping review was conducted on March ۵, ۲۰۲۵, with searches in Scopus (۲۶۳ records), Web of Science (۱۹۶ records), and PubMed (۱۹۵ records). From an initial pool of ۶۵۴ records, ۳۳۴ duplicates were removed, leaving ۳۱۹ records for screening. Exclusion criteria included non-research documents, irrelevant titles, abstracts, or full texts. Data from eligible studies were extracted and descriptively analyzed, categorized into national, specialty, and preclinical examinations. Results: Advanced models like GPT-۴ frequently surpassed passing thresholds in national examinations (e.g., USMLE: ۸۹.۹%, JMLE: ۸۱.۵%) and specialty examinations (e.g., ophthalmology: ۹۱%, orthopedics: ۷۳.۹%). However, weaknesses were evident in image-based questions (accuracy: ۴۵%-۷۸%), complex clinical reasoning, and local contexts. GPT-۴ consistently outperformed older models (e.g., GPT-۳.۵, Bard). Conclusion: AI chatbots demonstrate significant potential as educational tools in medicine, but limitations in multimodal processing and cultural adaptation persist. Future research should focus on improving these aspects and integrating chatbots into medical education frameworks.

کلیدواژه ها:

AI Chatbots ، LLMS ، Medical Examinations ، ChatGPT