Implementation of national eLogbook AI-BI dashboards for residents’ clinical performance assessment

Background and aims: logbooks are purposeful assessment tools to record the learner’s clinicalperformance. They facilitate self-reflection, self-assessment and helps medical residents toachieve their educational goals. In this study we aimed to design the logbook Artificial Intelligence-Business Intelligence (AI-BI) dashboard based on Integrated data from eLogbook systemsin all universities of medical sciences with the aim of the national policy makers, residency programboard members, academic managers to make intelligent decisions about the residents learningand assessing process.Method: This applied cross sectional study was conducted by the Information Technology workinggroup of the Education deputy at the Ministry of Health and Medical Education in ۱۴۰۱. Medicaluniversities that hold medical residency programs were included to the study. the Universitiesthat use the electronic logbook system were identified. The ETL technique was used to integratelogbook data from different universities sources into a national warehouse platform. To buildintelligent dashboard at first thematic analysis was performed to explore the effective key performanceindicators (KPI) for national monitoring of medical resident performance on logbooks.Based on the result of thematic analysis, a questionnaire was developed to conduct a Delphimethod to rank and consensus the KPI metrics. Then we used OLAP engine to create cubes andprocess KPIs from logbooks data panel.Results: ۴۴ out of ۶۷ medical universities hold residency programs. They use four eLogbookdifferent products for the medical resident clinical assessment. The data in the following threeyears (۲۰۱۹ and ۲۰۲۳) from all universities databases integrated into a national eLogbook warehouseto build an AI-BI enabled dashboard. The results of the thematic analysis and the twoDelphi round finally led to selection of seven main KPIs for analysis and reporting resident academicachievement at the national level. eLogbook dashboard was deployed based classificationand deep learning algorithms and used to visualize multidimensional data about residents. deeplearning algorithms predicted the resident academic achievement with an AUC of ۰.۸۲. Machinelearning classifiers (SVM and RF) trained to predict residents’ progress in in- training examinations.THE deep learning algorithm had the highest performance (AUC, ۰.۷۴).Conclusion: In this study we designed and implemented the artificial intelligence dashboard toreport medical resident’s performance on logbooks in the whole country. AI-BI logbook dashboardhas provided the ability to gain insight about learning process and make reform to increaseresident’ learning achievements in different medical specialties. Deep learning algorithms aremore reliable and accurate for predicting student performance.

کلیدواژه ها:

artificial intelligence ، business intelligence ، electronic logbook ، medical resident ، medical university ، dashboard