Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Evaluation of Virtual Reality Bronchoscopy as a Learning and Assessment Tool

Evaluation of Virtual Reality Bronchoscopy as a Learning and Assessment Tool IntroductionTraining in invasive procedures is traditionally based on a master-apprentice model. Trainees gain competence in a particular procedure through supervised training. The risk with the ‘see one, do one, teach one’ model is of patients being victims to the inexperience of the trainee. This is of greater relevance in endoscopy than surgery, as the patients are very often awake and only mildly sedated. Training has thus moved from the traditional operating and endoscopy suites to simulation laboratories. The aviation industry was among the first to realise the potential of simulation in training. Training in a flight simulator is nowadays mandatory for pilots of commercial and military airplanes. The airline industry has also shown a transfer-effectiveness ratio of 50%. This means that 1 h spent in the simulator saves half an hour in air [1].Virtual reality (VR) is the combination of human-computer interfaces, graphics, sensor technology, high-end computing and networking to allow a user to become immersed in and interact with an artificial environment [2]. Rapid advances in computers in terms of computing speed and graphics have led to the development of other endoscopic [3, 4]and surgical simulators [5].With increasing scrutiny of the medical profession’s performance, there is a need to make objective assessment an essential component of training. The incorporation of assessment in training is a form of quality assurance in the future [6]. Assessment as a form of feedback allows training in a structured manner as well as focussed attention on areas of concern during the performance of a task. Assessment in endoscopy is largely subjective. According to a survey of bronchoscopists, 50 procedures were considered necessary to be judged competent [7]. There are as yet no objective methods of assessment in bronchoscopy. VR-based teaching models such as the VR bronchoscopy simulators have the potential to address this issue.It is however essential to validate any system of assessment prior to its incorporation into a training programme. The aim of this study was to establish the face, construct and content validity of the VR bronchoscopy simulator and also to evaluate the utility of the system as a learning aid. Face validity is defined as a measure of how appropriate the test is to real-life, for the purposes of assessment. Construct validity is the extent to which the assessment tool reflects the concept that is being tested. Content validity is the extent to which the tool is representative of the knowledge or the skill that is being tested [8].Materials and MethodsVR SimulatorThe bronchoscopic simulator (HT Medical Systems, Gaithersburg, Md., USA) consists of a computer programme which runs on Windows NT 4.0 with Service Pack 5 on a Pentium III, 500 MHz CPU with 128MB (fig. 1). There is a flexible bronchoscope, which is inserted into the nasal passage of a dummy face on the system. The system provides a realistic image of the tracheo-bronchial tree as the user navigates through the programme. The system’s realistic simulation is augmented by the fact that there are facilities for suction and that the vocal cords move with each breath. There is a provision for local anaesthetic spray and image capture. Inadequate anaesthetic use results in the system simulating a cough, with its associated consequences of scope tip movement and displacement. There are a number of case simulations where each case is unique in terms of the number and the character of the pathologies. On identification of the pathology, the users are asked to capture the image which is recorded by the system.Fig. 1VR bronchoscopy simulator.Study GroupsNine novices without previous bronchoscopic experience formed the study group (group 1). Nine experienced bronchoscopists having performed between 200 and 1,000 bronchoscopies formed the other group (group 2).Study Design, Training and FeedbackInformed consent was obtained from the subjects in both groups. The subjects from group 1 were given an opportunity to get familiar with the endoscope and the anatomy by a session unlimited by time on one of the cases, which did not have any identifiable pathology. The availability of road signs (fig. 2) during this ‘practice’ case aided the subjects in understanding the anatomy. Each subject then undertook 7–10 sessions on the simulator. At the end of each session the subjects were given a feedback of their performance by allowing them to view the computer-generated assessment sheet. Each subject from group 2 undertook 2 sessions. The first session, on a case with no demonstrable pathology, gave them the opportunity to familiarise themselves with the system. The parameters as recorded by the system on the second case, formed the data for this study. The system’s face validity was established by asking the subjects of this group to answer a questionnaire consisting of their opinion on the accuracy of simulation and the quality of the graphics, the training utility of the simulator, the reliability of the system’s parameters in assessing competence and their recommendations on the addition of any other parameters.Fig. 2Tracheo-bronchial segments with ‘road signs’.Methods of Assessment (Study Data)The simulator has the ability to assess competence by a set of parameters, which are displayed at the end of each case. These are the percentage of segments visualised, the number of wall collisions, time taken to complete the examination, the order in which the segments are studied and the amount of local anaesthetic used. The first three parameters formed the data for the study. The percentage of segments seen is a reflection of the completeness of the procedure, while economy of performance, total time taken and number of wall collisions are indicators of dexterity or technical skill.Data AnalysisThe first attempt of group 1 was compared to the performance of group 2. We assessed the efficacy of the system as a learning tool by studying whether there was a significant difference between the first and subsequent sessions of subjects from group 1 in terms of the percentage of segments visualised, economy of performance and the number of wall collisions. Each attempt of the novices was compared with the data from the experienced group to study the point at which there was no significant difference between the two groups.Comparison between the two groups was done using the Mann-Whitney U test. The comparison between the sessions, of subjects from group 1, was done using the Wilcoxon signed ranks test.ResultsFrom table 1 it can be noted that when the first attempt of the novices (group 1) was compared with the skilled bronchoscopists (group 2), the subjects in group 2 visualised a significantly larger percentage of bronchial segments with a significantly lower number of wall collisions and a better economy of performance.Table 1Attempt 1 of group 1 vs. group 2There was a significant improvement in the percentage of segments visualised by the third attempt (p = 0.05). There was significant reduction in the number of wall collisions (p = 0.02) and an improvement in the economy of performance (p = 0.008) by the sixth attempt. Even though there was a reduction in the total time taken to perform the procedure between the fifth and the sixth, this did not achieve significance.There was no difference in any of the parameters when the fifth attempt of the subjects of group 1 was compared with those from group 2 (table 2). The significance in the difference between the two groups in terms of the percentage of segments visualised (p = 0.09) and in the economy (p = 0.06) disappeared by the third attempt. The significance in the difference in the number of wall collisions disappeared in the fifth attempt (p = 0.06).Table 2Attempt 5 of group 1 vs. group 2In response to the questionnaire, all the subjects from group 2 commented on the accuracy of the anatomical simulation, though one of the respondents felt that the right upper lobe bronchus came off too close to the carina. They judged the parameters to be reliable and considered the simulator to be an effective training tool. A majority of them were of the opinion that further studies would be needed to demonstrate a correlation between performance on the simulator and performance in real procedures.DiscussionSpeed and accuracy are of prime importance in performing a bronchoscopy as the procedure could result in considerable discomfort to the patient, especially when performed by a trainee. The need to train novices in the procedure prior to performance of a real procedure has been emphasized before [9]. Synthetic models (Storz, UK and Denoyer-Geppart Int., Chicago, Ill., USA) are available in most bronchoscopy suites to enable the trainee to become familiar with the functions of the bronchoscope and the tracheobronchial anatomy. The disadvantages of these are the relative rigidity of the models, the unreal tissue quality and lack of accurate simulation [9]. With the use of animals, other than the ethical problems involved [10], there are considerable anatomical differences [9].Training in bronchoscopy, as in surgery, is random and there is no way of ensuring that all trainees in bronchoscopy are exposed to a wide spectrum of cases and problems [2]. VR training devices allow standardised progression from simple to more complex tasks. The tasks are repeatable and can be optimised to a trainee’s needs [11]. It is thus possible to create a structured goal-based training programme and to judge the point at which a trainee achieves adequate expertise to carry out a bronchoscopy on a patient.Currently, training in bronchoscopy is dependent on a number of factors such as the commitment of the trainer, the co-operation of the patient and availability of the endoscopy suite and staff. The free availability of the VR bronchoscope in a training centre addresses all these factors. While the VR bronchoscope cannot replace the trainer-trainee-based teaching model, it can aid in shortening the learning curve.Endoscopic training based on the conventional master-apprentice model is also expensive. An analysis of the cost involved in training residents in gastrointestinal endoscopy in the endoscopy suite revealed that the training by this method has considerable financial implications [12]. This is especially relevant in today’s environment of managed care and financial cutbacks.From this study we have been able to establish the value of the simulator as a learning tool. It is fair to assume that the percentage of the segments visualised is an indicator of the knowledge of the tracheobronchial anatomy and that the economy of performance and the number of wall collisions are an indicator of dexterity. As it would be logically expected, while knowledge was acquired fairly early, it took a few more sessions on the simulator to develop dexterity. An early grasp of knowledge was a result of the training aids that the subjects used and the feedback they received at the end of every session.Even though the total time taken to complete the procedure improved, the difference did not achieve significance. This is because with the acquisition of anatomical orientation, the novices were able to study the whole tracheobronchial tree. It is likely that with further attempts there would have been a significant reduction in the time taken for the subjects to complete the examination. After 4 h of training on the simulator, Colt et al. [13]found an improvement in the accuracy of performance, but did not find an improvement in the speed (time taken) and the percentage of time spent in red-out. Ost et al. [14], found that while there was a significant improvement in terms of the time taken for the procedure and the percentage of segments visualised between the first five attempts and the next five attempts, the difference in terms of the percentage of time spent in red-out was achieved later. The conclusion from these two other studies and our study is that training on the simulator helps in shortening the learning curve fairly early in terms of the accuracy of the procedure but time taken (speed) and dexterity take longer to master.In this study we have been able to show the improvement in the performance of the novice group with self-learning using only the aids available with the simulator, as compared to Colt et al. [13], who showed improvement on the simulator after a training session that consisted of supervised training on the simulator combined with video-based training. While VR simulators have a role in self-learning, it is important to note that other studies using lower gastrointestinal endoscopy [4]simulators have demonstrated the importance of supervised training on the simulator to aid the trainee in learning various manoeuvres which are required for the successful completion of a procedure. The fact that trainees improved on the simulator with only self-learning is probably explained by the fact that skills and knowledge required for bronchoscopy and lower gastrointestinal endoscopy are probably different due to the rigid structure of the tracheo-bronchial tree. Future studies on the simulator should try to evaluate the utility of the simulator as a self-learning tool and to show a transfer of skills to real-life procedures. Ost et al. [14]have shown a successful transfer of skills from VR to real bronchoscopies in a small number of subjects.From this study we have been able to establish the face, construct and content validity of the simulator as an assessment tool. The validity of the system has been established by other studies as well [13, 14]. The availability of the simulator as a valid assessment tool would be beneficial to the trainee as a form of objective feedback of performance. The use of VR simulators as a form of summative assessment would still be premature and further studies are needed.Further studies would also be required to translate the effect of learning on the simulator to real-life bronchoscopy. It would then be possible to demonstrate a transfer-effectiveness ratio.AcknowledgementsWe would like to thank HT Medical Systems, Gaithersburg, Md., USA for allowing us to use one of their systems for this study. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Respiration Karger

Evaluation of Virtual Reality Bronchoscopy as a Learning and Assessment Tool

Moorthy, K.; Smith, S.; Brown, T.; Bann, S.; Darzi, A.
Respiration , Volume 70 (2): 5 – Apr 1, 2003
Download PDF
5 pages

Loading next page...
 
/lp/karger/evaluation-of-virtual-reality-bronchoscopy-as-a-learning-and-NkJAIcHwwz

References (15)

Publisher
Karger
Copyright
© 2003 S. Karger AG, Basel
ISSN
0025-7931
eISSN
1423-0356
DOI
10.1159/000070067
Publisher site
See Article on Publisher Site

Abstract

IntroductionTraining in invasive procedures is traditionally based on a master-apprentice model. Trainees gain competence in a particular procedure through supervised training. The risk with the ‘see one, do one, teach one’ model is of patients being victims to the inexperience of the trainee. This is of greater relevance in endoscopy than surgery, as the patients are very often awake and only mildly sedated. Training has thus moved from the traditional operating and endoscopy suites to simulation laboratories. The aviation industry was among the first to realise the potential of simulation in training. Training in a flight simulator is nowadays mandatory for pilots of commercial and military airplanes. The airline industry has also shown a transfer-effectiveness ratio of 50%. This means that 1 h spent in the simulator saves half an hour in air [1].Virtual reality (VR) is the combination of human-computer interfaces, graphics, sensor technology, high-end computing and networking to allow a user to become immersed in and interact with an artificial environment [2]. Rapid advances in computers in terms of computing speed and graphics have led to the development of other endoscopic [3, 4]and surgical simulators [5].With increasing scrutiny of the medical profession’s performance, there is a need to make objective assessment an essential component of training. The incorporation of assessment in training is a form of quality assurance in the future [6]. Assessment as a form of feedback allows training in a structured manner as well as focussed attention on areas of concern during the performance of a task. Assessment in endoscopy is largely subjective. According to a survey of bronchoscopists, 50 procedures were considered necessary to be judged competent [7]. There are as yet no objective methods of assessment in bronchoscopy. VR-based teaching models such as the VR bronchoscopy simulators have the potential to address this issue.It is however essential to validate any system of assessment prior to its incorporation into a training programme. The aim of this study was to establish the face, construct and content validity of the VR bronchoscopy simulator and also to evaluate the utility of the system as a learning aid. Face validity is defined as a measure of how appropriate the test is to real-life, for the purposes of assessment. Construct validity is the extent to which the assessment tool reflects the concept that is being tested. Content validity is the extent to which the tool is representative of the knowledge or the skill that is being tested [8].Materials and MethodsVR SimulatorThe bronchoscopic simulator (HT Medical Systems, Gaithersburg, Md., USA) consists of a computer programme which runs on Windows NT 4.0 with Service Pack 5 on a Pentium III, 500 MHz CPU with 128MB (fig. 1). There is a flexible bronchoscope, which is inserted into the nasal passage of a dummy face on the system. The system provides a realistic image of the tracheo-bronchial tree as the user navigates through the programme. The system’s realistic simulation is augmented by the fact that there are facilities for suction and that the vocal cords move with each breath. There is a provision for local anaesthetic spray and image capture. Inadequate anaesthetic use results in the system simulating a cough, with its associated consequences of scope tip movement and displacement. There are a number of case simulations where each case is unique in terms of the number and the character of the pathologies. On identification of the pathology, the users are asked to capture the image which is recorded by the system.Fig. 1VR bronchoscopy simulator.Study GroupsNine novices without previous bronchoscopic experience formed the study group (group 1). Nine experienced bronchoscopists having performed between 200 and 1,000 bronchoscopies formed the other group (group 2).Study Design, Training and FeedbackInformed consent was obtained from the subjects in both groups. The subjects from group 1 were given an opportunity to get familiar with the endoscope and the anatomy by a session unlimited by time on one of the cases, which did not have any identifiable pathology. The availability of road signs (fig. 2) during this ‘practice’ case aided the subjects in understanding the anatomy. Each subject then undertook 7–10 sessions on the simulator. At the end of each session the subjects were given a feedback of their performance by allowing them to view the computer-generated assessment sheet. Each subject from group 2 undertook 2 sessions. The first session, on a case with no demonstrable pathology, gave them the opportunity to familiarise themselves with the system. The parameters as recorded by the system on the second case, formed the data for this study. The system’s face validity was established by asking the subjects of this group to answer a questionnaire consisting of their opinion on the accuracy of simulation and the quality of the graphics, the training utility of the simulator, the reliability of the system’s parameters in assessing competence and their recommendations on the addition of any other parameters.Fig. 2Tracheo-bronchial segments with ‘road signs’.Methods of Assessment (Study Data)The simulator has the ability to assess competence by a set of parameters, which are displayed at the end of each case. These are the percentage of segments visualised, the number of wall collisions, time taken to complete the examination, the order in which the segments are studied and the amount of local anaesthetic used. The first three parameters formed the data for the study. The percentage of segments seen is a reflection of the completeness of the procedure, while economy of performance, total time taken and number of wall collisions are indicators of dexterity or technical skill.Data AnalysisThe first attempt of group 1 was compared to the performance of group 2. We assessed the efficacy of the system as a learning tool by studying whether there was a significant difference between the first and subsequent sessions of subjects from group 1 in terms of the percentage of segments visualised, economy of performance and the number of wall collisions. Each attempt of the novices was compared with the data from the experienced group to study the point at which there was no significant difference between the two groups.Comparison between the two groups was done using the Mann-Whitney U test. The comparison between the sessions, of subjects from group 1, was done using the Wilcoxon signed ranks test.ResultsFrom table 1 it can be noted that when the first attempt of the novices (group 1) was compared with the skilled bronchoscopists (group 2), the subjects in group 2 visualised a significantly larger percentage of bronchial segments with a significantly lower number of wall collisions and a better economy of performance.Table 1Attempt 1 of group 1 vs. group 2There was a significant improvement in the percentage of segments visualised by the third attempt (p = 0.05). There was significant reduction in the number of wall collisions (p = 0.02) and an improvement in the economy of performance (p = 0.008) by the sixth attempt. Even though there was a reduction in the total time taken to perform the procedure between the fifth and the sixth, this did not achieve significance.There was no difference in any of the parameters when the fifth attempt of the subjects of group 1 was compared with those from group 2 (table 2). The significance in the difference between the two groups in terms of the percentage of segments visualised (p = 0.09) and in the economy (p = 0.06) disappeared by the third attempt. The significance in the difference in the number of wall collisions disappeared in the fifth attempt (p = 0.06).Table 2Attempt 5 of group 1 vs. group 2In response to the questionnaire, all the subjects from group 2 commented on the accuracy of the anatomical simulation, though one of the respondents felt that the right upper lobe bronchus came off too close to the carina. They judged the parameters to be reliable and considered the simulator to be an effective training tool. A majority of them were of the opinion that further studies would be needed to demonstrate a correlation between performance on the simulator and performance in real procedures.DiscussionSpeed and accuracy are of prime importance in performing a bronchoscopy as the procedure could result in considerable discomfort to the patient, especially when performed by a trainee. The need to train novices in the procedure prior to performance of a real procedure has been emphasized before [9]. Synthetic models (Storz, UK and Denoyer-Geppart Int., Chicago, Ill., USA) are available in most bronchoscopy suites to enable the trainee to become familiar with the functions of the bronchoscope and the tracheobronchial anatomy. The disadvantages of these are the relative rigidity of the models, the unreal tissue quality and lack of accurate simulation [9]. With the use of animals, other than the ethical problems involved [10], there are considerable anatomical differences [9].Training in bronchoscopy, as in surgery, is random and there is no way of ensuring that all trainees in bronchoscopy are exposed to a wide spectrum of cases and problems [2]. VR training devices allow standardised progression from simple to more complex tasks. The tasks are repeatable and can be optimised to a trainee’s needs [11]. It is thus possible to create a structured goal-based training programme and to judge the point at which a trainee achieves adequate expertise to carry out a bronchoscopy on a patient.Currently, training in bronchoscopy is dependent on a number of factors such as the commitment of the trainer, the co-operation of the patient and availability of the endoscopy suite and staff. The free availability of the VR bronchoscope in a training centre addresses all these factors. While the VR bronchoscope cannot replace the trainer-trainee-based teaching model, it can aid in shortening the learning curve.Endoscopic training based on the conventional master-apprentice model is also expensive. An analysis of the cost involved in training residents in gastrointestinal endoscopy in the endoscopy suite revealed that the training by this method has considerable financial implications [12]. This is especially relevant in today’s environment of managed care and financial cutbacks.From this study we have been able to establish the value of the simulator as a learning tool. It is fair to assume that the percentage of the segments visualised is an indicator of the knowledge of the tracheobronchial anatomy and that the economy of performance and the number of wall collisions are an indicator of dexterity. As it would be logically expected, while knowledge was acquired fairly early, it took a few more sessions on the simulator to develop dexterity. An early grasp of knowledge was a result of the training aids that the subjects used and the feedback they received at the end of every session.Even though the total time taken to complete the procedure improved, the difference did not achieve significance. This is because with the acquisition of anatomical orientation, the novices were able to study the whole tracheobronchial tree. It is likely that with further attempts there would have been a significant reduction in the time taken for the subjects to complete the examination. After 4 h of training on the simulator, Colt et al. [13]found an improvement in the accuracy of performance, but did not find an improvement in the speed (time taken) and the percentage of time spent in red-out. Ost et al. [14], found that while there was a significant improvement in terms of the time taken for the procedure and the percentage of segments visualised between the first five attempts and the next five attempts, the difference in terms of the percentage of time spent in red-out was achieved later. The conclusion from these two other studies and our study is that training on the simulator helps in shortening the learning curve fairly early in terms of the accuracy of the procedure but time taken (speed) and dexterity take longer to master.In this study we have been able to show the improvement in the performance of the novice group with self-learning using only the aids available with the simulator, as compared to Colt et al. [13], who showed improvement on the simulator after a training session that consisted of supervised training on the simulator combined with video-based training. While VR simulators have a role in self-learning, it is important to note that other studies using lower gastrointestinal endoscopy [4]simulators have demonstrated the importance of supervised training on the simulator to aid the trainee in learning various manoeuvres which are required for the successful completion of a procedure. The fact that trainees improved on the simulator with only self-learning is probably explained by the fact that skills and knowledge required for bronchoscopy and lower gastrointestinal endoscopy are probably different due to the rigid structure of the tracheo-bronchial tree. Future studies on the simulator should try to evaluate the utility of the simulator as a self-learning tool and to show a transfer of skills to real-life procedures. Ost et al. [14]have shown a successful transfer of skills from VR to real bronchoscopies in a small number of subjects.From this study we have been able to establish the face, construct and content validity of the simulator as an assessment tool. The validity of the system has been established by other studies as well [13, 14]. The availability of the simulator as a valid assessment tool would be beneficial to the trainee as a form of objective feedback of performance. The use of VR simulators as a form of summative assessment would still be premature and further studies are needed.Further studies would also be required to translate the effect of learning on the simulator to real-life bronchoscopy. It would then be possible to demonstrate a transfer-effectiveness ratio.AcknowledgementsWe would like to thank HT Medical Systems, Gaithersburg, Md., USA for allowing us to use one of their systems for this study.

Journal

Respiration – Karger

Published: Apr 1, 2003

Keywords: Endoscopy and training; Assessment and endoscopy/surgery; Virtual reality and training; Bronchoscopy

There are no references for this article.