Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

E-learning - evaluation of a diagnostic decision support system in undergraduate medical training

E-learning - evaluation of a diagnostic decision support system in undergraduate medical training Background: Enhanced diagnostic reasoning by medical students has a positive effect on learning diagnostic procedures. Decision support systems (DSS) are intended to support this process in the clinical phase of medical education. The aim was to explore both the use and usefulness of such a system, as well as adjustments needed for an optimal implementation in undergraduate medical training. Methods: Using a dispersive implementation strategy, fourth-year medical students (n=130) were given access to a DSS (DXplain) during the two-semester course in internal medicine. The students were asked to participate in an internet-based questionnaire survey at the end Results: A majority (61.8%) of the users found the system more or less useful, 41.8% found it relatively easy to use, and a majority (54.9%) would like to have future access to the system. However, the usage was limited, and several issues needed for an optimal implementation were found related to interface, logic, usefulness, accessibility and motivation. Conclusion: The DXplain system were more or less useful and relatively easy to use in undergraduate medical training, and the usage was limited using an dispersive implementation. An optimal implementation needs to be stricter including taking several aspects of usefulness into account, and a pronounced integration in theoretical and clinical phases of the education is needed to take advantage of its full potential in undergraduate medical training. KEYWORDS: e-learning, undergraduate medical training, decision support system, usefulness. Introduction At the frontier of medical education, increased attention has been directed towards modern educational methods such as e-learning [1]. An example of e-learning is a computer based decision support system (DSS). A DSS is intended to facilitate learning and feedback for students in the clinical phase of their education as well as to be a support in continuous use in the work-up of their integrative assignments. These assignments are based upon clinical health issues and constitute symptoms of both common and rare diseases. A DSS could also serve as a unifying link throughout the medical education program. DXplain is a DSS that was developed at Massachusetts General Hospital and Harvard Medical School and used since the mid 1980s, and it has the characteristics of both an electronic medical textbook and a medical reference system [2]. The system covers more than 4900 clinical manifestations and 2200 diagnoses. Various manifestations linked together provide a weighted probability of a series of differential diagnoses; by entering presented symptoms and manifestations, the system assesses the ten most likely diagnoses to comprise such a symptomatology. DXplain also offers an electronic medical reference system in which the diagnoses and associated medical history and clinical features are described [2], [3], [4]. Today DXplain is available online rather than as distributed software, with the advantages of easy access and simplified updating. In theoretical studies, DXplain and other DSS have been shown to be potent instruments for differential diagnostics. The differential diagnoses suggested by DSS have, to a large extent, corresponded well with those suggested by medical specialists [5], [6]. A recent study indicated that DXplain is highly cost-effective in clinical settings [7]. However, further studies are necessary in order to obtain a good overview of the usefulness of DSS in clinical settings [8], [9], [7]. DXplain has proven to be an appreciated aid for medical students during their internal medicine residency [10], and other studies have shown that students with access to Iliad, another DSS, perform better and score higher on diagnostic tests than students without such access [11], [12]. An increase in differential diagnostic reasoning has positive effects on learning diagnostic procedures. Broad-minded thinking from the start and keeping multiple differential diagnoses in mind increases versatility and the ability to adapt and to adjust the treatment strategy in case of shifting diagnostic developments. At the same time, this approach reduces the risk of malpractice by health care providers focusing on an erroneous diagnosis. Medical professionals often subconsciously anchor and adjust the symptoms of a patient to agree more with an early preliminary diagnosis, which is known as anchoring and adjustment bias or confirmation bias [13]. The aim of this project was to explore the use and perceived usefulness of a DSS, as well as to determine adjustments needed for an optimal implementation of a DSS in undergraduate medical training, using a dispersive implementation approach. Materials and Methods Setting and Participants Karolinska Institute is the largest medical school in Sweden. A new curriculum has been developed with increased focus on integration of theoretical and practical knowledge based on 107 integrative tasks. Starting in the third year, teaching is clinically-based, and takes place at hospitals and in primary health care. One hundred and thirty third-year students were given access to the DXplain system during their one-year course in internal medicine at two university hospitals. The course spans two semesters and consists of theoretical education for three consecutive weeks, followed by rotations of three weeks of clinical training on different wards and in primary health care. In addition, sub-courses in infectious diseases and dermatology are included in the course. At the end of each semester there are five weeks of elective courses. Implementation was of a dispersive nature; the students were given instructions on how to access and use the system. Access was gained through a proxy link after entering the trial user name and password as well as the e-mail address and title of the user. During the evaluation period reminders were sent to all participants by e-mail. Questionnaire At the end of the course all students were asked to participate in an internetbased questionnaire survey consisting of 13 closed-ended questions, each of which had five options and a comment field. The questionnaire was transla- ted and modified from a questionnaire used by Bauer, B.A. et al. in 2002. The answers were given a more standardised ordinal scale of multiple choice options (e.g. ranging from Very hard to Very easy, Never to Always, etc.). The questionnaire had an additional comment field in which participants were encouraged to further elaborate on their experiences using approximately half a page (Appendix). Qualitative analysis of open-ended questions In order to assemble and summarise the comments given in the questionnaire, these were analysed anonymously in a qualitative content analysis [14], [15]. First, all comments were read in order to get a general impression of the text and its contents. After that the meaning units were identified and decontextualised from the manifest content of the apparent text. These were abstracted and categorised or coded into thematic codes. The meaning units were condensed and recoded into conceptual codes and sub-categorical paragraphs from the latent content of the underlying meaning of the text. This process of recoding was repeated until a summary of concepts and phenomena could be recontextualised in accordance with the original text. A verification of the final analysis was done by returning to the original text, as a measure for ensuring the coherence of the analysis and the primary general impression. The verification serves to consolidate the validity and affirm the concepts of the analysis, ensuring that the original text and its meaning are not lost or distorted in the process. The analysis is a dynamic process, which must be iterated repeatedly with the original text as a constant archetype to enable a representative and conclusive image of what the text conveys. Thus, the process of coding the latent content is the most important part of the analysis, the intention of which is to bring an inceptive chaos into order. Usage The log files of the usage of DXplain from the KI trial account were analysed; the provided e-mail addresses were linked to individual students, which allowed for mapping of the usage of DXplain in terms of time and quantity. Microsoft Office Excel was used for statistics and diagrams. Results Answer rates Of the 130 students who were asked to complete the questionnaire, 102 answered it properly (78.5%). Fifty of them had started the course in the autumn of 2008, while 52 had started in the spring of 2009. Of the 102 students, 59 were women and 43 were men. Of the 55 who used DXplain (responding "seldom" or "once or twice a week"), 28 were men and 27 were women. Otherwise the results showed no significant differences between the genders. Frequency of use Concerning the usage of DXplain, 55 (53.9%) of the responders (n=102) answered that they had used the system at some point (Figure 1); 48 (87.3%) had used it ,,Seldom and seven (12.7%) had used it ,,Once or twice per week. Thirty-five (63.6%) of these users (n=55) had used the system only during the theoretical phase of their studies, two (3.6%) only during clinical rotations, and 18 (32.7%) during both. The frequency of use was also lower during clinical rotations. Non-users 60 50 40 30 20 10 0 0 Several times per day 0 Daily 7 Once or tw ice per w eek 11 2 Seldom Never 35 47 Clin (2) Clin + Th (18) Th (35) As seen in log files, a total of 70 (53.8%, n=130) students had used DXplain. Forty-five of the 55 students who answered that they had used the system, and 11 students who answered that they had never used it, could be verified to have been logged on. Another seven users could be identified, but did not answer the questionnaire, and another six users could not be identified. Usefulness Of the users, 34 (61.8%, n=55) found the system somewhat useful (Figure 2). The perceived usefulness did not differ between theoretical and clinical use. Responders (n=102) Figure 1. How often have you used DXplain? Responders (n=55) se fu l ot us ef ul N us ef ul Figure 2. Has DXplain been useful? Ease of use A majority of responders found the system easy to use (41.8%, n=55) or were indifferent (40%) (Figure 3). To most responders the system was neither easier nor harder to use than expected. Responders (n=55) le ar ly Ve ry W or th le ss 2 Very hard us ef ul 0 Very easy Easy Neither Hard Figure 3. How easy was it to use DXplain? Continuous access A 54.9% (n=102) majority of responders wanted continuous access to the system and 34.3% were undecided, leaving 10.8% who did not want continuous access. (Figure 4). Users (n = 55) 40 35 Non-Users (n = 47) Responders (n=102) 30 25 20 15 10 5 0 Very much Yes Undecided No 16 19 14 1 5 4 1 Not at all 11 15 16 Figure 4. Would you like to have continuous access to DXplain? Open-ended questions on optimal implementation Five main categories or codes could be discerned from the content of the comments by the users. These were: 1) Interface, 2) Logic, 3) Usefulness, 4) Accessibility and 5) Motivation (Figure 5) An additional sixth Non-users category could be established based on the comments of non-users regarding their usage of DXplain. Figure 5. Five main categories, discerned from the content of the open free-text answers. Interface Few students commented on the interface of DXplain; one student found it hard to use, while two others found it easy to understand and use. The lack of negative comments on interface issues (from otherwise negatively inclined users) suggests that the interface is not incomprehensible nor are attributes a problem regarding usage of the system. However, at least nine users encountered difficulties with the interface of the system, almost invariably with the entry of manifestations. It is not possible to enter laboratory results by entering figures directly, but only in verbal descriptions. This was exacerbated by the English medical vernacular, since English is not their native language, and by the fact that the students were unfamiliar with abbreviations for lab tests. This is illustrated by the following quote: "Somewhat inhibiting that you can't enter `MCHC decreased', because then there are no hits. It would have been better if you could search a lab test list, since you often have lab results available. Now you have to think of the sought-after English term and look up special nouns for everything." Logic The logic engine of DXplain was found to be rational and consistent in its "black and white" position on individual symptoms. This means that an entered symptom not in accord with the correct diagnosis completely alters the list of suggested diagnoses in the wrong direction. It also means that comorbidity or common symptoms (e.g. fatigue) become difficult to diagnose. Consequently, the system has high sensitivity, but low specificity, which is compensated for by the listing of several diagnoses, thus making it probable that the correct diagnosis is among the top ten. "During the internal medicine course there is much talk about broad diagnoses with many `common' symptoms ­ and if you then use DXplain, the result is a long list of possible diagnoses. And then the program hasn't been of much use to you at all. I haven't felt that DXplain has added much at all. The things that were good were the questions with suggested signs or symptoms that you can address and the account of signs and symptoms, which correspond more or less well to different suggested diagnoses." The logic lacks transparency due to an opaque account of the probability algorithms. That leads to difficulties in how to relate to the suggested diagnoses when it is unclear which manifestations affect the results in which direction. Ancillary symptoms then become impossible to ignore and it becomes hard to focus on cardinal symptoms. Thus, the user gets numerous suggested diagnoses without mutual relevance, ostensibly yielding only a draft of random diagnoses that does little to advance diagnostics. Again, the specificity is low, which is elaborated as follows by a student: "I can't see the pros with DXplain's algorithm, since it isn't documented or transparently explained when it comes to searches. The ranking is impossible to evaluate and then the result is just a list of possible diagnoses to explore further. This can be more easily done in documents and books you have at your disposal." Usefulness Dxplain was perceived as a useful aid for differential diagnostics and a good tool for obtaining a quick list of probable diagnoses that helps the user get on the right track and confirms the users initial train of thought. However, since unusual sets of symptoms constitute more difficult cases, the use of and need for a DSS is quantitatively limited. At the same time, the need is qualitatively limited for cases that are difficult due to an insufficient or unspecific basis of signs and symptoms (e.g. fatigue) even for common diseases. For difficult cases, DXplain was nevertheless perceived by the students as a good aid in diagnostics. The following quote illustrates this: "I think a DSS should be used more extensively in medical practice in the future since there is a large and ever increasing number of rare diseases not taught in our medical training. Integration of a DSS as a natural part of everyday medical practice would increase the quality of our work. Medical school is the best place for introduction of a DSS so new generations of doctors can see this tool as a natural part of their practice." One student commented that DXplain provides nothing on the treatment or pathophysiology of the diagnoses. The system lacks a next step after the identification of a diagnosis, which renders it an incomplete aid for training in internal medicine. Nevertheless, students who found it desirable maintained that it was beneficial in theory and they welcomed DSS as a part of their education. Accessibility Some 20 users described difficulties in accessing DXplain, as they considered it complicated and time-consuming to log on to the system. According to the users this was mainly due to the proxy procedure for distance access, a forgettable web site address and log-in data, and a tedious log-in procedure. To gain access the student needed the proxy settings for KI, the web site address and log-in data to be readily available, and lastly it was necessary to enter the data in two steps. Individually, these steps were seen as obstacles and together they were seen as one long and complicated procedure. Although students saw the benefits of using the system, the obstacles held them back. Students thought it took too long to reach the system and that there was little time to fully understand it. There was no lack of time per se, but the access procedure was inhibiting and gave a sense that there was indeed a lack of time­ particularly due to proxy settings. The students wrote that when they faced a problem, they needed to get information quickly and did not want to sit by a computer for a prolonged period of time. Due to an arduous log-in procedure students tried to be time-efficient and therefore used other literary alternatives. Motivation Nine users were explicitly indifferent to DXplain: there was a lack of incentive even to use the system. The dispersive implementation did not promote the system as something necessary and did not trigger their interest enough. One student complained about the introduction, where the focus was on practical fundamentals. Instead of a passive walk-through of how to use the system, there was a wish for an interactive introduction and examples of how and when it could and should be used. Furthermore, more evidence-based background information concerning the usefulness of diagnostic support systems was desired. The introduction was not sufficient for the students to remember how to use, or even to find the time to use, DXplain. They received little encouragement to use it for either the theoretical or clinical phases of the internal medicine course. More active promotion from teachers and tutors was advocated in order to remind students that such a system was at their disposal. As the students were not committed or motivated enough to use the system there was no natural stimulus, and they recommended active encouragement from teachers and tutors. Non-users The non-users generally showed less commitment and motivation and many commented that they had forgotten that the system even existed. This was not a token of disinterest, but rather that the students were not committed and motivated enough to cross the threshold and give the system a try. Most students mentioned an interest in DSS and its theoretical usefulness and said they would like to have future access to a DSS. The following quote is representative: "I have not used DXplain. I might do it in the future, but the reason I didn't use it this time is probably because we get so much material to go through that another `thing' to have to look at comes second, alas. I think it sounds very useful though, but due to a lack of time I haven't used it. Will we be able to access it with an available password, etc., in the future?" Discussion In this project we explored the use, perceived usefulness, and various aspects of an implementation of the DSS DXplain in a course in internal medicine. In summary, the system was used more often for theoretical studies than during clinical rotations. While usage was relatively low, it was used by a small group of early adaptors, even though they felt that they received little encouragement. The system was perceived as useful by most during the internal medicine course, but with limitations. Furthermore, a majority of users and non-users reported that they would like to have access to the system during the rest of the course, the following courses and continuously afterwards. Our qualitative approach revealed five main categories related to optimising the implementation, as well as specific issues such as an initial indifference to the system when given only access. The greatest strength lies in the qualitative analysis of the comments on the questionnaire, from which a large amount of information could be extracted from a relatively small population. Another strength of this work is the high response rate of 78.5%, supporting the reliability of the results. In order to estimate the reliability of the questionnaire results, the answers were checked for joint discrepancies. An important limitation is the integration strategy you have chosen, as the integration strategy is crucial for the use of a learning resource by students. Related research in clinical settings points out that the implementation is of utmost importance and require careful planning in the success of a DSS system [16], [17]. The usage of DXplain during the evaluation period was lower than expected due to little motivation, possibly originating in the students lack of genuine motivation to using a new diagnostic tool during their first clinical course. This was pronounced in combination with the need to learn a substantial amount of factual information during the internal medicine course, when the students experienced time constraints in using an extra learning aid.Instead, using the system seemed to have low priority compared to other more familiar sources of information like books and handouts. Another conspicuous problem was poor accessibility and perceived low usefulness which are both highly likely factors to influences the usage of a DSS. The students wanted to have quick access to medical references and accordingly saw little time effectiveness in using a DSS in favour of other sources, as similarly concluded of other forms of e-learning [17]. Despite the limited usage of DXplain during the evaluation, the students found the system to be a useful aid and fairly easy to use, and a majority, even among non-users, were interested in future access to a DSS. A DSS is a potent tool for bridging the unavoidable gaps in knowledge that are missed or not covered by the curriculum, both for students and doctors. Diagnoses not included in the curriculum comprise above all the rare and unusual diagnoses, and thus represent a small but important application area for a DSS. For more common diagnoses a DSS may not be a necessity for either doctors or advanced students, but should instead constitute a security measure against the human factor, when probable or even correct diagnoses are missed in a stress- ful environment. As the students had little, if any, experience regarding diagnoses in other specialties and were not required to deal with those diagnoses, they would naturally not see the potential of a DSS in considering the often perplexing array of symptoms of common diagnoses not covered by the internal medicine curriculum. If a DSS such as DXplain should become an important part of student learning activities, a more integrated implementation into the curiculum seems to be crucial, as with other forms of e-learning [17]. It should be introduced early and used at several stages both as a textbook and a reference system in an integrated way including both theoretical teaching, clinical training, and assessment. This would make the DSS a more accepted part of both studies and clinical work, instead of being a foreign modernity, and hence pave the way to a natural usage. Because of the redundancy for well-known diagnoses, an early introduction is fundamental, before students have learned to know and recognise such diagnoses without considering important differentials. In spite of the potential seen by the students, they also found several adjustments needed for an optimal implementation. Among the five main categories (interface, logic, usefulness, accessibility and motivation) accessibility seems to be the most prominent. To facilitate better accessibility to DXplain, a quick link with pre-set proxy settings has been tested, leaving only the login procedure. Encouragement from educators and reminders of both DXplain as well as its log-in data would also contribute to remembering and using the system and would thus affect the perceived accessibility. A demand for computer based solutions would force solutions for easy access and usefulness. A simplified interface and log-in would serve that purpose well, as the log-in affects the general impression of the system very early on. To work around the interface issues and other shortcomings that a DSS might have, it seems like time, motivation and educational resources are crucial. An optimal implementation is needed in order to teach students to use the software in a correct and practical manner, as a supplement in clinical training and diagnostic reasoning. Integation of a DSS into other e-learning systems could possibly improve student usage and engagement, such as recent advances in game-based learning [18]. In a future study on DXplain in undergraduate medical education it would be interesting to use of a more integrated scenario. For example giving the students concrete assignments to do using the system, or integrating its use in the assessment, would increase the usage and make it possible to study related factors in more detail. Conclusion After receiving a brief introduction and given access to DXplain, the usage was limited among undergraduate medical students. A majority of the users found the system more or less useful and relatively easy to use, with virtually no difference between theoretical and clinical usefulness. A majority of users and non-users would like to have future access to the system. The study explored several issues of use for an optimal implementation including related to interface, logic, usefulness, accessibility and motivation. Specific issues were an initial indifference to the system when given only access, difficulties with the interface, lack of transparency of the system, its usefulness as an aid for differential diagnostics, and the complicated log-on procedure. A stricter implementation of a diagnostic support system, and a pronounced integration in theoretical and clinical phases of the education is needed to take advantage of its full potential in undergraduate medical training. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Bio-Algorithms and Med-Systems de Gruyter

E-learning - evaluation of a diagnostic decision support system in undergraduate medical training

Download PDF
19 pages

Loading next page...
 
/lp/de-gruyter/e-learning-evaluation-of-a-diagnostic-decision-support-system-in-ARsrjWeJMu
Publisher
de Gruyter
Copyright
Copyright © 2012 by the
ISSN
1895-9091
eISSN
1896-530X
DOI
10.2478/bams-2012-0019
Publisher site
See Article on Publisher Site

Abstract

Background: Enhanced diagnostic reasoning by medical students has a positive effect on learning diagnostic procedures. Decision support systems (DSS) are intended to support this process in the clinical phase of medical education. The aim was to explore both the use and usefulness of such a system, as well as adjustments needed for an optimal implementation in undergraduate medical training. Methods: Using a dispersive implementation strategy, fourth-year medical students (n=130) were given access to a DSS (DXplain) during the two-semester course in internal medicine. The students were asked to participate in an internet-based questionnaire survey at the end Results: A majority (61.8%) of the users found the system more or less useful, 41.8% found it relatively easy to use, and a majority (54.9%) would like to have future access to the system. However, the usage was limited, and several issues needed for an optimal implementation were found related to interface, logic, usefulness, accessibility and motivation. Conclusion: The DXplain system were more or less useful and relatively easy to use in undergraduate medical training, and the usage was limited using an dispersive implementation. An optimal implementation needs to be stricter including taking several aspects of usefulness into account, and a pronounced integration in theoretical and clinical phases of the education is needed to take advantage of its full potential in undergraduate medical training. KEYWORDS: e-learning, undergraduate medical training, decision support system, usefulness. Introduction At the frontier of medical education, increased attention has been directed towards modern educational methods such as e-learning [1]. An example of e-learning is a computer based decision support system (DSS). A DSS is intended to facilitate learning and feedback for students in the clinical phase of their education as well as to be a support in continuous use in the work-up of their integrative assignments. These assignments are based upon clinical health issues and constitute symptoms of both common and rare diseases. A DSS could also serve as a unifying link throughout the medical education program. DXplain is a DSS that was developed at Massachusetts General Hospital and Harvard Medical School and used since the mid 1980s, and it has the characteristics of both an electronic medical textbook and a medical reference system [2]. The system covers more than 4900 clinical manifestations and 2200 diagnoses. Various manifestations linked together provide a weighted probability of a series of differential diagnoses; by entering presented symptoms and manifestations, the system assesses the ten most likely diagnoses to comprise such a symptomatology. DXplain also offers an electronic medical reference system in which the diagnoses and associated medical history and clinical features are described [2], [3], [4]. Today DXplain is available online rather than as distributed software, with the advantages of easy access and simplified updating. In theoretical studies, DXplain and other DSS have been shown to be potent instruments for differential diagnostics. The differential diagnoses suggested by DSS have, to a large extent, corresponded well with those suggested by medical specialists [5], [6]. A recent study indicated that DXplain is highly cost-effective in clinical settings [7]. However, further studies are necessary in order to obtain a good overview of the usefulness of DSS in clinical settings [8], [9], [7]. DXplain has proven to be an appreciated aid for medical students during their internal medicine residency [10], and other studies have shown that students with access to Iliad, another DSS, perform better and score higher on diagnostic tests than students without such access [11], [12]. An increase in differential diagnostic reasoning has positive effects on learning diagnostic procedures. Broad-minded thinking from the start and keeping multiple differential diagnoses in mind increases versatility and the ability to adapt and to adjust the treatment strategy in case of shifting diagnostic developments. At the same time, this approach reduces the risk of malpractice by health care providers focusing on an erroneous diagnosis. Medical professionals often subconsciously anchor and adjust the symptoms of a patient to agree more with an early preliminary diagnosis, which is known as anchoring and adjustment bias or confirmation bias [13]. The aim of this project was to explore the use and perceived usefulness of a DSS, as well as to determine adjustments needed for an optimal implementation of a DSS in undergraduate medical training, using a dispersive implementation approach. Materials and Methods Setting and Participants Karolinska Institute is the largest medical school in Sweden. A new curriculum has been developed with increased focus on integration of theoretical and practical knowledge based on 107 integrative tasks. Starting in the third year, teaching is clinically-based, and takes place at hospitals and in primary health care. One hundred and thirty third-year students were given access to the DXplain system during their one-year course in internal medicine at two university hospitals. The course spans two semesters and consists of theoretical education for three consecutive weeks, followed by rotations of three weeks of clinical training on different wards and in primary health care. In addition, sub-courses in infectious diseases and dermatology are included in the course. At the end of each semester there are five weeks of elective courses. Implementation was of a dispersive nature; the students were given instructions on how to access and use the system. Access was gained through a proxy link after entering the trial user name and password as well as the e-mail address and title of the user. During the evaluation period reminders were sent to all participants by e-mail. Questionnaire At the end of the course all students were asked to participate in an internetbased questionnaire survey consisting of 13 closed-ended questions, each of which had five options and a comment field. The questionnaire was transla- ted and modified from a questionnaire used by Bauer, B.A. et al. in 2002. The answers were given a more standardised ordinal scale of multiple choice options (e.g. ranging from Very hard to Very easy, Never to Always, etc.). The questionnaire had an additional comment field in which participants were encouraged to further elaborate on their experiences using approximately half a page (Appendix). Qualitative analysis of open-ended questions In order to assemble and summarise the comments given in the questionnaire, these were analysed anonymously in a qualitative content analysis [14], [15]. First, all comments were read in order to get a general impression of the text and its contents. After that the meaning units were identified and decontextualised from the manifest content of the apparent text. These were abstracted and categorised or coded into thematic codes. The meaning units were condensed and recoded into conceptual codes and sub-categorical paragraphs from the latent content of the underlying meaning of the text. This process of recoding was repeated until a summary of concepts and phenomena could be recontextualised in accordance with the original text. A verification of the final analysis was done by returning to the original text, as a measure for ensuring the coherence of the analysis and the primary general impression. The verification serves to consolidate the validity and affirm the concepts of the analysis, ensuring that the original text and its meaning are not lost or distorted in the process. The analysis is a dynamic process, which must be iterated repeatedly with the original text as a constant archetype to enable a representative and conclusive image of what the text conveys. Thus, the process of coding the latent content is the most important part of the analysis, the intention of which is to bring an inceptive chaos into order. Usage The log files of the usage of DXplain from the KI trial account were analysed; the provided e-mail addresses were linked to individual students, which allowed for mapping of the usage of DXplain in terms of time and quantity. Microsoft Office Excel was used for statistics and diagrams. Results Answer rates Of the 130 students who were asked to complete the questionnaire, 102 answered it properly (78.5%). Fifty of them had started the course in the autumn of 2008, while 52 had started in the spring of 2009. Of the 102 students, 59 were women and 43 were men. Of the 55 who used DXplain (responding "seldom" or "once or twice a week"), 28 were men and 27 were women. Otherwise the results showed no significant differences between the genders. Frequency of use Concerning the usage of DXplain, 55 (53.9%) of the responders (n=102) answered that they had used the system at some point (Figure 1); 48 (87.3%) had used it ,,Seldom and seven (12.7%) had used it ,,Once or twice per week. Thirty-five (63.6%) of these users (n=55) had used the system only during the theoretical phase of their studies, two (3.6%) only during clinical rotations, and 18 (32.7%) during both. The frequency of use was also lower during clinical rotations. Non-users 60 50 40 30 20 10 0 0 Several times per day 0 Daily 7 Once or tw ice per w eek 11 2 Seldom Never 35 47 Clin (2) Clin + Th (18) Th (35) As seen in log files, a total of 70 (53.8%, n=130) students had used DXplain. Forty-five of the 55 students who answered that they had used the system, and 11 students who answered that they had never used it, could be verified to have been logged on. Another seven users could be identified, but did not answer the questionnaire, and another six users could not be identified. Usefulness Of the users, 34 (61.8%, n=55) found the system somewhat useful (Figure 2). The perceived usefulness did not differ between theoretical and clinical use. Responders (n=102) Figure 1. How often have you used DXplain? Responders (n=55) se fu l ot us ef ul N us ef ul Figure 2. Has DXplain been useful? Ease of use A majority of responders found the system easy to use (41.8%, n=55) or were indifferent (40%) (Figure 3). To most responders the system was neither easier nor harder to use than expected. Responders (n=55) le ar ly Ve ry W or th le ss 2 Very hard us ef ul 0 Very easy Easy Neither Hard Figure 3. How easy was it to use DXplain? Continuous access A 54.9% (n=102) majority of responders wanted continuous access to the system and 34.3% were undecided, leaving 10.8% who did not want continuous access. (Figure 4). Users (n = 55) 40 35 Non-Users (n = 47) Responders (n=102) 30 25 20 15 10 5 0 Very much Yes Undecided No 16 19 14 1 5 4 1 Not at all 11 15 16 Figure 4. Would you like to have continuous access to DXplain? Open-ended questions on optimal implementation Five main categories or codes could be discerned from the content of the comments by the users. These were: 1) Interface, 2) Logic, 3) Usefulness, 4) Accessibility and 5) Motivation (Figure 5) An additional sixth Non-users category could be established based on the comments of non-users regarding their usage of DXplain. Figure 5. Five main categories, discerned from the content of the open free-text answers. Interface Few students commented on the interface of DXplain; one student found it hard to use, while two others found it easy to understand and use. The lack of negative comments on interface issues (from otherwise negatively inclined users) suggests that the interface is not incomprehensible nor are attributes a problem regarding usage of the system. However, at least nine users encountered difficulties with the interface of the system, almost invariably with the entry of manifestations. It is not possible to enter laboratory results by entering figures directly, but only in verbal descriptions. This was exacerbated by the English medical vernacular, since English is not their native language, and by the fact that the students were unfamiliar with abbreviations for lab tests. This is illustrated by the following quote: "Somewhat inhibiting that you can't enter `MCHC decreased', because then there are no hits. It would have been better if you could search a lab test list, since you often have lab results available. Now you have to think of the sought-after English term and look up special nouns for everything." Logic The logic engine of DXplain was found to be rational and consistent in its "black and white" position on individual symptoms. This means that an entered symptom not in accord with the correct diagnosis completely alters the list of suggested diagnoses in the wrong direction. It also means that comorbidity or common symptoms (e.g. fatigue) become difficult to diagnose. Consequently, the system has high sensitivity, but low specificity, which is compensated for by the listing of several diagnoses, thus making it probable that the correct diagnosis is among the top ten. "During the internal medicine course there is much talk about broad diagnoses with many `common' symptoms ­ and if you then use DXplain, the result is a long list of possible diagnoses. And then the program hasn't been of much use to you at all. I haven't felt that DXplain has added much at all. The things that were good were the questions with suggested signs or symptoms that you can address and the account of signs and symptoms, which correspond more or less well to different suggested diagnoses." The logic lacks transparency due to an opaque account of the probability algorithms. That leads to difficulties in how to relate to the suggested diagnoses when it is unclear which manifestations affect the results in which direction. Ancillary symptoms then become impossible to ignore and it becomes hard to focus on cardinal symptoms. Thus, the user gets numerous suggested diagnoses without mutual relevance, ostensibly yielding only a draft of random diagnoses that does little to advance diagnostics. Again, the specificity is low, which is elaborated as follows by a student: "I can't see the pros with DXplain's algorithm, since it isn't documented or transparently explained when it comes to searches. The ranking is impossible to evaluate and then the result is just a list of possible diagnoses to explore further. This can be more easily done in documents and books you have at your disposal." Usefulness Dxplain was perceived as a useful aid for differential diagnostics and a good tool for obtaining a quick list of probable diagnoses that helps the user get on the right track and confirms the users initial train of thought. However, since unusual sets of symptoms constitute more difficult cases, the use of and need for a DSS is quantitatively limited. At the same time, the need is qualitatively limited for cases that are difficult due to an insufficient or unspecific basis of signs and symptoms (e.g. fatigue) even for common diseases. For difficult cases, DXplain was nevertheless perceived by the students as a good aid in diagnostics. The following quote illustrates this: "I think a DSS should be used more extensively in medical practice in the future since there is a large and ever increasing number of rare diseases not taught in our medical training. Integration of a DSS as a natural part of everyday medical practice would increase the quality of our work. Medical school is the best place for introduction of a DSS so new generations of doctors can see this tool as a natural part of their practice." One student commented that DXplain provides nothing on the treatment or pathophysiology of the diagnoses. The system lacks a next step after the identification of a diagnosis, which renders it an incomplete aid for training in internal medicine. Nevertheless, students who found it desirable maintained that it was beneficial in theory and they welcomed DSS as a part of their education. Accessibility Some 20 users described difficulties in accessing DXplain, as they considered it complicated and time-consuming to log on to the system. According to the users this was mainly due to the proxy procedure for distance access, a forgettable web site address and log-in data, and a tedious log-in procedure. To gain access the student needed the proxy settings for KI, the web site address and log-in data to be readily available, and lastly it was necessary to enter the data in two steps. Individually, these steps were seen as obstacles and together they were seen as one long and complicated procedure. Although students saw the benefits of using the system, the obstacles held them back. Students thought it took too long to reach the system and that there was little time to fully understand it. There was no lack of time per se, but the access procedure was inhibiting and gave a sense that there was indeed a lack of time­ particularly due to proxy settings. The students wrote that when they faced a problem, they needed to get information quickly and did not want to sit by a computer for a prolonged period of time. Due to an arduous log-in procedure students tried to be time-efficient and therefore used other literary alternatives. Motivation Nine users were explicitly indifferent to DXplain: there was a lack of incentive even to use the system. The dispersive implementation did not promote the system as something necessary and did not trigger their interest enough. One student complained about the introduction, where the focus was on practical fundamentals. Instead of a passive walk-through of how to use the system, there was a wish for an interactive introduction and examples of how and when it could and should be used. Furthermore, more evidence-based background information concerning the usefulness of diagnostic support systems was desired. The introduction was not sufficient for the students to remember how to use, or even to find the time to use, DXplain. They received little encouragement to use it for either the theoretical or clinical phases of the internal medicine course. More active promotion from teachers and tutors was advocated in order to remind students that such a system was at their disposal. As the students were not committed or motivated enough to use the system there was no natural stimulus, and they recommended active encouragement from teachers and tutors. Non-users The non-users generally showed less commitment and motivation and many commented that they had forgotten that the system even existed. This was not a token of disinterest, but rather that the students were not committed and motivated enough to cross the threshold and give the system a try. Most students mentioned an interest in DSS and its theoretical usefulness and said they would like to have future access to a DSS. The following quote is representative: "I have not used DXplain. I might do it in the future, but the reason I didn't use it this time is probably because we get so much material to go through that another `thing' to have to look at comes second, alas. I think it sounds very useful though, but due to a lack of time I haven't used it. Will we be able to access it with an available password, etc., in the future?" Discussion In this project we explored the use, perceived usefulness, and various aspects of an implementation of the DSS DXplain in a course in internal medicine. In summary, the system was used more often for theoretical studies than during clinical rotations. While usage was relatively low, it was used by a small group of early adaptors, even though they felt that they received little encouragement. The system was perceived as useful by most during the internal medicine course, but with limitations. Furthermore, a majority of users and non-users reported that they would like to have access to the system during the rest of the course, the following courses and continuously afterwards. Our qualitative approach revealed five main categories related to optimising the implementation, as well as specific issues such as an initial indifference to the system when given only access. The greatest strength lies in the qualitative analysis of the comments on the questionnaire, from which a large amount of information could be extracted from a relatively small population. Another strength of this work is the high response rate of 78.5%, supporting the reliability of the results. In order to estimate the reliability of the questionnaire results, the answers were checked for joint discrepancies. An important limitation is the integration strategy you have chosen, as the integration strategy is crucial for the use of a learning resource by students. Related research in clinical settings points out that the implementation is of utmost importance and require careful planning in the success of a DSS system [16], [17]. The usage of DXplain during the evaluation period was lower than expected due to little motivation, possibly originating in the students lack of genuine motivation to using a new diagnostic tool during their first clinical course. This was pronounced in combination with the need to learn a substantial amount of factual information during the internal medicine course, when the students experienced time constraints in using an extra learning aid.Instead, using the system seemed to have low priority compared to other more familiar sources of information like books and handouts. Another conspicuous problem was poor accessibility and perceived low usefulness which are both highly likely factors to influences the usage of a DSS. The students wanted to have quick access to medical references and accordingly saw little time effectiveness in using a DSS in favour of other sources, as similarly concluded of other forms of e-learning [17]. Despite the limited usage of DXplain during the evaluation, the students found the system to be a useful aid and fairly easy to use, and a majority, even among non-users, were interested in future access to a DSS. A DSS is a potent tool for bridging the unavoidable gaps in knowledge that are missed or not covered by the curriculum, both for students and doctors. Diagnoses not included in the curriculum comprise above all the rare and unusual diagnoses, and thus represent a small but important application area for a DSS. For more common diagnoses a DSS may not be a necessity for either doctors or advanced students, but should instead constitute a security measure against the human factor, when probable or even correct diagnoses are missed in a stress- ful environment. As the students had little, if any, experience regarding diagnoses in other specialties and were not required to deal with those diagnoses, they would naturally not see the potential of a DSS in considering the often perplexing array of symptoms of common diagnoses not covered by the internal medicine curriculum. If a DSS such as DXplain should become an important part of student learning activities, a more integrated implementation into the curiculum seems to be crucial, as with other forms of e-learning [17]. It should be introduced early and used at several stages both as a textbook and a reference system in an integrated way including both theoretical teaching, clinical training, and assessment. This would make the DSS a more accepted part of both studies and clinical work, instead of being a foreign modernity, and hence pave the way to a natural usage. Because of the redundancy for well-known diagnoses, an early introduction is fundamental, before students have learned to know and recognise such diagnoses without considering important differentials. In spite of the potential seen by the students, they also found several adjustments needed for an optimal implementation. Among the five main categories (interface, logic, usefulness, accessibility and motivation) accessibility seems to be the most prominent. To facilitate better accessibility to DXplain, a quick link with pre-set proxy settings has been tested, leaving only the login procedure. Encouragement from educators and reminders of both DXplain as well as its log-in data would also contribute to remembering and using the system and would thus affect the perceived accessibility. A demand for computer based solutions would force solutions for easy access and usefulness. A simplified interface and log-in would serve that purpose well, as the log-in affects the general impression of the system very early on. To work around the interface issues and other shortcomings that a DSS might have, it seems like time, motivation and educational resources are crucial. An optimal implementation is needed in order to teach students to use the software in a correct and practical manner, as a supplement in clinical training and diagnostic reasoning. Integation of a DSS into other e-learning systems could possibly improve student usage and engagement, such as recent advances in game-based learning [18]. In a future study on DXplain in undergraduate medical education it would be interesting to use of a more integrated scenario. For example giving the students concrete assignments to do using the system, or integrating its use in the assessment, would increase the usage and make it possible to study related factors in more detail. Conclusion After receiving a brief introduction and given access to DXplain, the usage was limited among undergraduate medical students. A majority of the users found the system more or less useful and relatively easy to use, with virtually no difference between theoretical and clinical usefulness. A majority of users and non-users would like to have future access to the system. The study explored several issues of use for an optimal implementation including related to interface, logic, usefulness, accessibility and motivation. Specific issues were an initial indifference to the system when given only access, difficulties with the interface, lack of transparency of the system, its usefulness as an aid for differential diagnostics, and the complicated log-on procedure. A stricter implementation of a diagnostic support system, and a pronounced integration in theoretical and clinical phases of the education is needed to take advantage of its full potential in undergraduate medical training.

Journal

Bio-Algorithms and Med-Systems – de Gruyter

Published: Jan 1, 2012

There are no references for this article.