Assessment of the Capability of ChatGPT-۳.۵ in Medical Physiology Examination in an Indian Medical School

Background: There has been increasing interest in exploring the capabilities of artificial intelligence (AI) in various fields, including education. Medical education is an area where AI can potentially have a significant impact, especially in helping students answer their customized questions. In this study, we aimed to investigate the capability of ChatGPT, a conversational AI model in generating answers to medical physiology exam questions in an Indian medical school.Methods: This cross-sectional study was conducted in March ۲۰۲۳ in an Indian Medical School, Deoghar, Jharkhand, India. The first mid-semester physiology examination was taken as the reference examination. There were two long essays, five short essay questions (total mark ۴۰), and ۲۰ multiple-choice questions (MCQ) (total mark ۱۰). We generated the response from ChatGPT (in March ۱۳ version) for both essay and MCQ questions. The essay-type answer sheet was evaluated by five faculties, and the average was taken as the final score. The score of ۱۲۵ students (all first-year medical students) in the examination was obtained from the departmental registery. The median score of the ۱۲۵ students was compared with the score of ChatGPT using Mann-Whitney U test.Results: The median score of ۱۲۵ students in essay-type questions was ۲۰.۵ (Q۱-Q۳: ۱۸-۲۳.۵) which corresponds to a median percentage of ۵۱.۲۵% (Q۱-Q۳: ۴۵-۵۸.۷۵) (P=۰.۱۴۷). The answer generated by ChatGPT scored ۲۱.۵ (Q۱-Q۳: ۲۱.۵-۲۲), which corresponds to ۵۳.۷۵% (Q۱-Q۳: ۵۳.۷۵-۵۵) (P=۰.۱۲۵). Hence, ChatGPT scored like that of the students (P=۰.۴) in essay-type questions. In MCQ-type questions, ChatGPT answered ۱۹ correctly in ۲۰ questions (score=۹.۵), and this was higher than the median score of students (۶) (Q۱-Q۳: ۵-۶.۵) (P<۰.۰۰۰۱).Conclusion: ChatGPT has the potential to generate answers to medical physiology examination questions. It has a higher capability to solve MCQ questions than essay-type ones. Although ChatGPT was able to provide answers that had the quality to pass the examination, the capability of generating high-quality answers for educational purposes is yet to be achieved. Hence, its usage in medical education for teaching and learning purposes is yet to be explored.

کلیدواژه ها:

Distance ، education ، Artificial Intelligence ، ChatGPT ، Physiology ، examination ، Students ، Medical