The role of artificial intelligence (AI) in medical education: A pilot study on ChatGPT’s impact on history-taking skills and objective structured clinical examination (OSCE) preparedness in undergraduate medical students
Article Sidebar
Main Article Content
Abstract
Background. Effective communication, particularly history-taking, is essential for medical professionals to ensure accurate diagnoses and treatment plans. Traditional training methods, such as role-playing and standardised patients, can be resource intensive. Advances in artificial intelligence (AI) offer new solutions, with ChatGPT, an advanced AI language model, capable of simulating patient interactions, providing scalable and flexible practice opportunities that enhance clinical skills training.
Objectives. This pilot study explores medical students’ perceptions of using ChatGPT as a virtual patient for history-taking practice and evaluates its potential impact on objective structured clinical examination (OSCE) preparedness by comparing pre- and post-intervention OSCE scores.
Methods. Two hundred second-year medical students were invited to participate; 40 students completed the post-practice survey. These participants used ChatGPT (versions 3.5 and 4.0) to practice history-taking using prompts aligned with the Calgary-Cambridge framework. The survey evaluated usability, realism, skill improvement, feedback quality, and satisfaction. Separately, OSCE performance data for the entire class from the 2023 cohort (pre-ChatGPT) and 2024 cohort (post-ChatGPT) were compared using a two-sample t-test to assess differences in history-taking scores. Qualitative survey responses were thematically analysed.
Results. Among survey respondents, 85% found ChatGPT easy to use, 75% rated the scenarios as realistic, and 80% reported improvements in history- taking skills. Additionally, 78% felt better prepared for their OSCE. A statistically significant 4% increase in OSCE history-taking scores was observed between the 2023 cohort (mean 39/50, 78%) and the 2024 cohort (mean 41/50, 82%; p<0.05). Qualitative feedback highlighted ChatGPT’s flexibility, ease of access, and immediate feedback, though students noted limitations in emotional nuance and occasional technical challenges.
Conclusion. The improvement in OSCE history-taking scores at a class-wide level suggests a potential association between ChatGPT use and enhanced clinical performance. While only a subset of students completed the survey, their responses indicate that ChatGPT is a valuable supplement to traditional training, particularly for self-directed practice. These findings support the integration of AI into clinical education to strengthen communication skills and OSCE preparedness. Future research should include broader participation and long-term evaluations to fully determine the role of AI in medical education.
Downloads
Article Details
Issue
Section
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The AJHPE is published under an Attribution-Non Commercial International Creative Commons Attribution (CC-BY-NC 4.0) License. Under this license, authors agree to make articles available to users, without permission or fees, for any lawful, non-commercial purpose. Users may read, copy, or re-use published content as long as the author and original place of publication are properly cited.
Exceptions to this license model is allowed for UKRI and research funded by organisations requiring that research be published open-access without embargo, under a CC-BY licence. As per the journals archiving policy, authors are permitted to self-archive the author-accepted manuscript (AAM) in a repository.
How to Cite
References
1. Kurtz SM, Silverman JD, Draper J. Teaching and Learning Communication Skills in Medicine. Boca Raton: CRC Press, 2016.
2. Fanning RM, Gaba DM. The role of debriefing in simulation-based learning. Simul Healthc 2007;2(2):115-125.
https://doi.org/ 10.1097/SIH.0b013e3180315539
3. Silverman J, Kurtz SM, Draper J. Skills for Communicating with Patients. Boca Raton: CRC Press, 2013.
4. Cook DA, Erwin PJ, Triola MM. Computerised virtual patients in health professions education: A systematic
review and meta-analysis. Acad Med 2010;85(10):1589-1602. https://doi.org/10.1097/ACM.0b013e3181edfe13 5. Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform 2018;22(5):1589-1604. https://doi.
org/10.1109/JBHI.2017.2767063
6. Ahmed Y. Utilisation of ChatGPT in medical education: Applications and implications for curriculum enhancement. Acta Inform Med 2023;31(4):300-305. https://doi.org/10.5455/aim.2023.31.300-305
7. Plackett R, Kassianos AP, Mylan S, et al. The effectiveness of using virtual patient educational tools to improve medical students’ clinical reasoning skills: A systematic review. BMC Med Educ 2022;22:365. https://doi. org/10.1186/s12909-022-03410-x
8. Wartman SA, Combs CD. Medical education must move from the information age to the age of artificial intelligence. Acad Med 2018;93(8):1107-1109. https://doi.org/10.1097/ACM.0000000000002044
9. CreswellJW.ResearchDesign:Qualitative,QuantitativeandMixedMethodApproaches,4thed.London:Sage,2014. 10. Kurtz SM, Silverman JD, Benson J, Draper J. Marrying content and process in clinical method teaching: Enhancing the Calgary-Cambridge guides. Acad Med 2003;78(8):802-809. https://doi.org/10.1097/00001888-
200308000-00011
11. Narayanan S, Ramakrishnan R, Durairaj E, et al. Artificial intelligence revolutionizing the field of medical education. Cureus 2023;15(11): e49604. https://doi.org/10.7759/cureus.49604
12. Cameron RS, Yong GP. The evolution and role of simulation in medical education. Anesthesia Patient Safety Foundation 2023. https://www.apsf.org/article/the-evolution-and-role-of-simulation-in-medical-education/ (accessed 22 July 2024).
13. Bienstock J, Heuer A. A review on the evolution of simulation-based training to help build a safer future. Medicine (Baltimore) 2022;101(25):e29503. https://doi.org/10.1097/MD.0000000000029503
14. Kruger D. Adaptive learning technology to enhance self-directed learning. In: Olivier J, ed. Self-directed multimodal learning in higher education 2020; (NWU Self-Directed Learning Series Volume 5) (pp. 93–116). AOSIS. https://doi.org/10.4102/aosis.2020.BK210.03
15. Kononowicz AA, Woodham LA, Edelbring S, et al. Virtual patient simulations in health professions education: Systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res 2019; 21(7)L:e14676. https://doi.org/10.2196/14676
16. Boud D, Molloy E. Feedback in Higher and Professional Education: Understanding It and Doing It Well. Abingdon-on-Thames: Routledge, 2013.
17. VanLehn, K. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ Psychol 2011;46(4):197-221.
18. Kirkpatrick DL, Kirkpatrick JD. Evaluating Training Programs: The Four Levels. Oakland, Calif.: Berrett-Koehler Publishers, 2006.
19. Tsang R. Practical applications of ChatGPT in undergraduate medical education. J Med Educ Curricular Dev 2023;10:23821205231178449 https://doi.org/10.1177/23821205231178449
20. McCoy L. Virtual Patient Simulations for Medical Education: Increasing Clinical Reasoning Skills through Deliberate Practice. Doctoral dissertation, Arizona State University, 2014. https://core.ac.uk/download/ pdf/79572495.pdf
21. Walter Y. Embracing the future of artificial intelligence in the classroom: The relevance of AI literacy, prompt engineering, and critical thinking in modern education. Int J Educ Techno Higher Educ 2024;21:15. https://doi. org/10.1186/s41239-024-00448-3
22. Zhu E, Lilienthal A, Shluzas LA, Masiello I, Zary N. Design of mobile augmented reality in health care education: A theory-driven framework. JMIR Med Educ 2015;1(2):e10. https://doi.org/10.2196/mededu.4443.
23. Li J, Zong H, Wu E, et al. Exploring the potential of artificial intelligence to enhance the writing of English academic papers by non-native English-speaking medical students - the educational application of ChatGPT. BMC Med Educ 2024; 24:736. https://doi.org/10.1186/s12909-024-05738-y
24. Elnahal SM, Blackford A, Smith K, et al. Identifying predictive factors for incident reports in patients receiving radiation therapy. Int J Radiat Oncol Biol Phys 2016;94(5):993-999. https://doi.org/10.1016/j.ijrobp.2015.11.041. 25. Consorti F, Mancuso R, Nocioni M, Piccolo A. Efficacy of virtual patients in medical education: A meta-analysis
of randomised studies. Computers Educ 2012;59(3):1001-1008.