Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/de-gruyter/development-and-evaluation-of-a-virtual-patient-based-exam-in-5K5PsPhkLC
- Publisher
- de Gruyter
- Copyright
- Copyright © 2015 by the
- ISSN
- 1895-9091
- eISSN
- 1896-530X
- DOI
- 10.1515/bams-2015-0008
- Publisher site
- See Article on Publisher Site
Abstract
Occupational medicine (OM), especially the ability to recognize potentially workplace-related diseases or accidents, is important for medical students to learn. The assessment in OM at the Ludwig-Maximilians-University is implemented as a paper-based, multiple-choice question exam that does not assess the students' ability to remember occupational aspects in clinical contexts. Therefore, the aim of this study was to create and evaluate an online exam based on 19 virtual patients to assess the aspects of OM in an interdisciplinary setting. Thirty-nine medical students participated in the exam with an average score of 65%. The score for freetext questions assessing OM aspects was low (39%). The study showed that the implementation of an interdisciplinary VP-based exam is feasible. The integration of such an interdisciplinary exam would be feasible, for example, as an open-book assessment. Keywords: interdisciplinary assessment; occupational medicine; virtual patient. Background Occupational medicine (OM) is a broad specialty dealing with the prevention and treatment of workplace accidents and workplace-associated diseases. The aspects of OM are important for almost all healthcare professionals. In particular, the ability to recognize a potentially workplace-related disorder, disease, or accident (including subsequent effects) and to initiate the necessary steps *Corresponding author: Inga Hege, Institut für Didaktik und Ausbildungsforschung in der Medizin, University Hospital of Munich (LMU), Ziemssenstr. 1, Munich 80337, Germany, E-mail: inga.hege@med.uni-muenchen.de Iris Schröpfer: Klinik für Anästhesiologie, University Hospital of Munich (LMU), Munich, Germany Katja Radon: Center for International Health at Institute and Outpatient Clinic for Occupational, Social and Environmental Medicine, University Hospital of Munich (LMU), Munich, Germany is crucial. Therefore, teaching OM in an undergraduate healthcare curriculum is of particular importance. However, there are two challenges in regards to the student perspective: a low interest in OM and a perceived low relevance to their medical practice. Often, real patients are not available for face-to-face teaching sessions [1, 2] because OM is typically organized as an outpatient clinic. The medical faculty at the Ludwig-MaximiliansUniversity (LMU) in Munich was one of the first faculties to integrate virtual patients (VPs) into the OM curriculum to overcome these challenges and enhance the engagement and interest of students in OM [3, 4]. VPs are "interactive computer simulations of real-life clinical scenarios for the purpose of medical training, education, or assessment [5]". The advantages of VPs are, among others, a standardization of teaching, availability (independence from time and place), training in a safe environment, and a learner-centered approach [6]. They have been widely implemented as learning activities but are also increasingly used as assessment tools in recent years in undergraduate healthcare curricula [7, 8]. The assessment in OM at the LMU is currently implemented as a traditional paper-based, multiple-choice (MC) question exam. This approach is efficient to assess learners' knowledge [9] but does not seem to be optimal to test the interdisciplinary aspects, as the exam topic itself is a strong cue and students prepare specifically for this exam. For example, in such a setting, it is not possible to test the students' ability to remember occupational aspects in other clinical contexts, a core (and minimum) learning objective that should be achieved during medical school. Therefore, the aim of this study was to develop and evaluate an online exam based on short VP cases that assess the aspects of OM in interdisciplinary settings without revealing the OM focus of the exam. Materials and methods VP creation The exam included 14 VPs containing one or two OM aspects and five VPs covering general internal medicine or surgery problems. 102Hege et al.: A virtual patient-based exam in occupational medicine The selection of the OM problems was done according to the following criteria: Frequency of recognized occupational diseases in Germany Learning objectives of the occupational medical curriculum at the LMU Well-balanced VP population concerning age (including children), gender, and occupation. The criteria have been determined in a discussion round with experts and teachers at the Institute and Outpatient Clinic for Occupational, Social and Environmental Medicine at the LMU and medical educators. Table 1 shows the topics of the VPs and the covered aspects that have been selected according to the criteria. The patient stories were either created based on (anonymized) healthcare records of the Institute and Outpatient Clinic for Occupational, Social and Environmental Medicine at the LMU or constructed based on the learning objectives. Where required, a written consent of the patients for the use of their (anonymized) data, such as radiology images or history, was obtained. Each VP consisted of four to six cards containing text, multimedia material, and questions (MC and freetext). The question had to be answered before proceeding to the next card. Navigating back was allowed, but it was not possible to change given answers. The process through the VP was similar for all cases: 1. Description of situation, first patient contact 2. Interpretation of findings and symptoms Table 1:Topics and problems of VPs. Topic Addressed OM aspect Procedure after an occupational accident, procedure after a needlestick injury Recognition of occupational aspects Recognition of workplace relation, notification of an occupational disease Occupational history taking Recognition of workplace relation Recognition of occupational aspects Notification of an occupational disease Obligation to report, information about mother protection law Recognition of noise as a risk factor Occupational history taking, occurrence of allergens Occupational history taking, notification of an occupational disease Obligation to report, notification of an occupational disease Notification of an occupational disease Occupational history taking, workplace ergonomics 3. Differential diagnoses 4. Management options The VPs were reviewed concerning content and didactical aspects and were evaluated by two medical students to ensure an appropriate level of difficulty and assess the time needed to complete the exam. For the creation, delivery, and analysis of the exam, the VP system CASUS® [10] was used. The system is used for all VP-based training throughout the medical curriculum at the LMU, so the students were familiar with the navigation. All relevant data such as time spent on a card, answers given, rating, and clicks were stored by the system. Rating Overall, the VP exam contained 72 questions, 24 of which assessed an OM aspect. Forty-four of the questions (8 with OM aspect) were followed by depending questions, so the correct answers were displayed to the student before proceeding to the next question; no solution was given for all others. The MC and freetext questions were assessed with one point if all options have been chosen correctly, otherwise zero. The answers were automatically rated by the system, and the automatic rating of freetext answers was reviewed and re-rated if necessary to ensure no correct answers were missed. Needlestick injury (hepatitis) Chronic fatigue syndrome Circulatory disorder due to vibrations Acute myocardial infarction Fracture of the femoral neck Silicosis Acute appendicitis Way-to-work accident Carpal canal syndrome Pleural mesothelioma Salmonellosis Hypertensive crisis Noise-induced hearing loss Contact eczema Herniated vertebral disk Tuberculosis Acute cholecystolithiasis Rupture of the cruciate ligament Tension headache and migraine Hege et al.: A virtual patient-based exam in occupational medicine103 Exam setting The participation for the 2 h exam was voluntary, and the participants were rewarded with a medical textbook. The exam was conducted for 2 days in a computer laboratory (20 workplaces) at the medical faculty. The students were told that they can participate in an interdisciplinary exam that could help them prepare for their final exam. A tutor was available to help with any technical and administrative issues during the exam. After the exam, a 12-item online questionnaire was displayed. Items could be rated on a Likert scale (1=totally agree, 5=totally disagree). The questionnaire consisted of personal questions: age, gender, clinical year, and interest in medical specialty general questions: whether they (and, if so, when) recognized the OM focus of the exam, self-estimation of knowledge and interest in OM, and estimation of the relevance of this specialty rating of the exam difficulty and comparison with traditional MC exams freetext comments concerning what was good about the exam, what could be improved, and other comments. The interest in OM was rated with a mean of 3.3 (SD=0.8), and the relevance of this specialty for physicians was rated with an average of 2.4 (SD=0.6) on a scale from 1 (totally agree) to 5 (totally disagree). Using the same scale, the level of difficulty of the exam was rated as appropriate (average=2.8, SD=0.5), and the participants preferred this type of exam to MC exams (average=2.4, SD=0.8). Twenty-six participants (67%) recognized that the focus of this exam were the aspects of OM, most of whom (n=20) discovered this during the first two VPs. The reasons were the re-occurrence of questions concerning the medical history and setting in which the workplace or occupation was mentioned. In the comments, the study participants mentioned that they liked the high realism of the VPs, the selection of topics, and the shortness of the VPs. They criticized that questions were formulated too open and that the quality of the radiology images was low. Exam results All 19 VPs were completed by all participants. The average session duration was 92 min. On average, 65% (44%77%) of the overall score was achieved. The mean success value for questions covering OM aspects was 44% (18%77%). There was no statistically significant difference of the success rate of MC versus freetext questions. However, for questions covering OM aspects, the success rate for the MC questions was statistically significantly higher than for freetext questions (Table 2). Noticeably, low success values were recorded for the following aspects: Asking patients about their profession (occupational history) Necessity of sending a notification to the Employers' Liability Insurance Association in the case of a workplace accident or occupational illness Table 2:Success rates of the questions. Question type Success Min, score, % % 65 44 68 69a 64 39a 44 18 54 33 39 14 Max, Number of % questions 77 77 84 100 79 71 72 16 23 17 48 43 Statistical analysis The session data of the exam and the questionnaire results were analyzed in SPSS. The relative frequencies, means, and SDs were calculated. Results Exam implementation Overall, 39 final-year medical students (23 female, 16 male) participated in the exam; the average age was 26.4 years (SD=1.8). No technical problems were encountered during the exam. The main interest of the participants was surgery (24%, n=7). None of the participants mentioned OM nor had chosen this specialty as a final-year elective. MC and freetext MC and freetext related to OM MC (all) MC related to OM Freetext (all) Freetext related to OM a p<0.05, statistically significant difference (success rate OM-related MC vs. OM-related freetext). 104Hege et al.: A virtual patient-based exam in occupational medicine Procedure when confronted by sequelae from previously work-related accidents or diseases. was quite high. If planning a "real" interdisciplinary exam (e.g., in OM and internal medicine), many more short VPs covering different internal medicine aspects would be required to increase the validity and reliability of such an exam. Third, the first VP students had to work through was a needlestick injury, a classic example for a workplace accident. Such obvious OM-related topics could be shifted toward the end of the exam. However, in a real exam setting, the order of the VPs would have to be permuted to make cheating more difficult. Although almost 70% of the participants recognized the occupational focus of the exam after working though the second VP, there was no increase of the success value for the subsequent VPs. Discussion The relevance, interest, and knowledge in OM was rated average, which is only slightly higher than in other surveys [11]. The participation in the exam was voluntary, so we assume that the participants were, in general, highly motivated, which might influence the results of such a study. However, as the focus of the study was not announced, we also assume that the students did not have a higher motivation or interest in OM. This assumption was confirmed by the results of the questionnaire. In contrast to the low performance, the participants rated the level of difficulty as appropriate. This confirms that the selection of topics and the design of the exam were reasonable. However, the participants did not know their overall score when filling out the survey and being confronted with a low score would presumably have influenced their of the level of difficulty. An unexpected result was that 69% of the participants had already recognized the OM focus of the exam after the second VP. We identified three potential reasons for this early detection: First, it seemed that the MC questions assessing OM aspects gave strong cues in particular. Freetext questions did not give these cues but are not very popular among students and often have to be reassessed by a teacher after a first automatic rating by a teacher (limiting the benefits of computer-based exams concerning time and resource saving). Therefore, it seems that long menu questions [12] could be a suitable answer format: The students type in a word and choose the appropriate option from a scroll-down menu. The list has to contain enough items that the student has to know the answer before choosing the appropriate item. This answer format minimizes cueing and can be rated automatically, but more research has to be undertaken concerning its reliability [13]. Second, the ratio of VPs covering OM aspects (n=14) versus general internal medicine or surgery aspects (n=5) Exam results Although the students stated that they recognized the OM focus of the exam very early on, their score for the subsequent VPs related to OM did not improve. Especially when asked in freetext format, which requires active knowledge, the success rate was low (38%). This low performance of the final-year students raises the question of how the acquisition of instructional knowledge in OM, especially in a primarily non-OM context, could be improved. This is important, as physicians are often confronted with patients having a medical problem related to an occupation outside an OM context. Outlook The study showed that the implementation of an interdisciplinary VP-based exam is feasible. With modifications, especially using long menu lists instead of MC questions, the exam will be subject to another study. The CASUS® software has already been proven to be a reliable exam software (e.g., the European School of Advanced Veterinary Studies and many US medical schools) that offers exam-specific features such as permutation of VPs, answer items, and security mechanisms for software or database failures. However, there are still barriers to overcome in regards to the implementation of regular summative exams in the curriculum at the LMU: Environment and equipment to run an online exam for 500 students per year Administrative effort to introduce such an interdisciplinary exam format into the examination regulations at the faculty and approval of all involved specialties. A more feasible way of introducing such an exam could be in the form of a formative open-book assessment [14]. The Hege et al.: A virtual patient-based exam in occupational medicine105 students can take the exam in a self-directed way, anytime and anywhere using textbooks, the Internet, or any other resource. Qualitative and quantitative immediate feedback, which was not provided during or after the study and is not provided for regular curricular exams, could be provided after having completed the exam. Feedback has a positive impact on student performance; however, the degree depends on the learner [15]. Such feedback could, for example, emphasize the consequences for patients if occupational aspects are not considered. To improve the performance in OM of the final-year students, we plan to adapt some of the VPs integrated into the curriculum of internal medicine and surgery to cover and repeat instructional OM aspects throughout the curriculum. Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission. Research funding: None declared. Employment or leadership: None declared. Honorarium: None declared. Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.
Journal
Bio-Algorithms and Med-Systems – de Gruyter
Published: Jun 15, 2015