Validation of a clinical rotation evaluation for physician assistant students

Validation of a clinical rotation evaluation for physician assistant students Background: We conducted a prospective validation study to develop a physician assistant (PA) clinical rotation evaluation (PACRE) instrument. The specific aims of this study were to 1) develop a tool to evaluate PA clinical rotations, and 2) explore associations between validated rotation evaluation scores and characteristics of the students and rotations. Methods: The PACRE was administered to rotating PA students at our institution in 2016. Factor analysis, internal consistency reliability, and associations between PACRE scores and student or rotation characteristics were determined. Results: Of 206 PACRE instruments sent, 124 were returned (60.2% response). Factor analysis supported a unidimensional model with a mean (SD) score of 4.31 (0.57) on a 5-point scale. Internal consistency reliability was excellent (Cronbach α=0.95). PACRE scores were associated with students’ gender (P = .01) and rotation specialty (P = .006) and correlated with students’ perception of being prepared (r = 0.32; P < .001) and value of the rotation (r =0.57; P < .001). Conclusions: This is the first validated instrument to evaluate PA rotation experiences. Application of the PACRE questionnaire could inform rotation directors about ways to improve clinical experiences. The findings of this study suggest that PA students must be adequately prepared to have a successful experience on their rotations. PA programs should consider offering transition courses like those offered in many medical schools to prepare their students for clinical experiences. Future research should explore whether additional rotation characteristics and educational outcomes are associated with PACRE scores. Keywords: Assessment, Clinical rotation, Physician assistant education, Validation study Background Data are sparse regarding the evaluation of training Physician assistants (PAs) are vital to all aspects of programs and individual clinical rotations in other edu- health care delivery. The number of PA training pro- cational settings. In medical school, students have evalu- grams continues to increase to meet the demand for ac- ated clerkships using electronic Likert-scaled checklists cess to health care [1–3]. In the past decade, the number [11]. In graduate medical education, residents have used of accredited PA programs and applicants to these pro- validated questionnaires to evaluate their programs with grams has increased dramatically [4, 5]. An integral part respect to rotation workload, faculty/learning environ- of these programs are clinical rotations, which are lim- ment, and stress [12–15]. Other studies have examined ited by competition and shortages [6–9]. The educa- resident assessments of programs and rotations in sur- tional quality of these clinical rotations can vary [10]. gery [16], internal medicine [17, 18], and anesthesiology Maintaining standards in clinical rotations, including [19]. Although PAs are trained in the medical model, validated assessment of performance, is a prerequisite to differences in clinical rotation length, content, supervi- ensuring a high-quality PA workforce. sion, and logistics make the use of existing clinical rota- tion evaluations less relevant to PA training settings. We are unaware of prior research on validated measures of * Correspondence: wittich.christopher@mayo.edu PA clinical rotations. Division of General Internal Medicine, Mayo Clinic, 200 First St. SW, Rochester, MN 55905, USA Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Meverden et al. BMC Medical Education (2018) 18:123 Page 2 of 7 To fill this gap, we conducted a prospective validation PACRE instrument development study to develop a PA clinical rotation evaluation A PACRE questionnaire was developed on the basis of (PACRE) instrument. The specific aims of this study existing literature [12, 14, 15, 20–28]. Items were devel- were to 1) develop a tool to evaluate PA clinical oped for each of the Stanford Faculty Development rotations, and 2) once validated, explore associations be- Program (SFDP) for Clinical Teaching categories: learn- tween PACRE rotation evaluation scores and character- ing climate, control of session, communication of goals, istics of the students and rotations. The purpose of the promotion of understanding and retention, evaluation, PACRE instrument was to determine and document a feedback, and promotion of self-directed learning [24, 29]. student’s perceptions of a rotation based on research on Two additional categories—rotation logistics and a rota- components of effective clinical teaching. We hypothe- tion global assessment—were included. After iterative re- sized that such an instrument would have strong in- vision, 2 items were selected for each of the 9 domains, ternal structure validity and scores would be associated for a total of 18 items in the PACRE instrument (Table 1). with rotation or student demographic variables. Responses were based on a 5-point Likert scale (1, strongly disagree, to 5, strongly agree). The final PACRE Methods instrument was pilot tested on 5 former PA students and Participants and clinical sites 5 current PA colleagues, which led to minor rewording of This prospective validation study involved PA students some items. who completed a clinical rotation in 2016 at1) the Mayo Clinic School of Health Sciences/Gundersen Medical Data collection and analysis Foundation/University of Wisconsin – La Crosse PA The PACRE was sent via an emailed link to each PA stu- Program or 2) the Mayo Clinic or the Mayo Clinic dent at the completion of the clinical rotation. Our sur- Health System from other PA programs. The study was vey research center managed the data collection using deemed exempt by the Mayo Clinic institutional review Qualtrics software (Qualtrics LLC). A reminder email board (Identification number: 15–006040). was sent 1 week after the completion of the rotation. Table 1 PACRE Instrument Educational Domains, Items, Item Loadings, and Mean Scores Educational Domain PACRE Item Item Mean (SD) Loading PACRE Score Learning climate The preceptor (s) created an environment that was conducive to learning. 0.7958 4.40 (0.75) The preceptor (s) was/were enthusiastic. 0.6585 4.56 (0.71) Control of session The preceptor (s) balanced time between patient care and teaching. 0.7751 4.35 (0.85) The preceptor (s) utilized my time effectively. 0.7515 4.24 (0.81) Communication of goals The rotation goals were stated clearly. 0.7937 4.20 (0.82) The rotation goals were appropriate for my educational needs. 0.7108 4.38 (0.69) Promotion of understanding The educational content was clearly communicated. 0.7536 4.33 (0.70) and retention The preceptor (s) helped to facilitate my understanding and retention 0.7770 4.47 (0.65) of information. Evaluation My performance was assessed by the preceptor (s). 0.8028 4.26 (0.72) I was evaluated on what I learned. 0.7788 4.08 (0.81) Feedback I received feedback on my performance. 0.8524 4.22 (0.75) The preceptor (s) communicated constructive assessments of my abilities. 0.8188 4.22 (0.79) Promotion of self-directed learning I had access to educational resources. 0.2183 4.68 (0.49) I was encouraged to learn on my own. 0.5270 4.54 (0.55) Rotation logistics I was oriented to information regarding rotation logistics. 0.7360 4.16 (0.75) My overall experience was organized. 0.7626 4.37 (0.68) Global assessment I would recommend this rotation. 0.7463 4.36 (0.82) I would recommend this site/work area as a place of employment. 0.6587 4.41 (0.75) Overall (16 items) 4.31 (0.57) Abbreviation: PACRE physician assistant clinical rotation evaluation Items from this educational domain loaded ambiguously and were excluded. Cronbach α (internal consistency reliability) was 0.95 for the 16 retained items. The eigenvalue for factor 1 was 9.63. Eigenvalues for all additional factors were < 1, which supports a one factor model for the PACRE instrument. Meverden et al. BMC Medical Education (2018) 18:123 Page 3 of 7 Demographic characteristics were collected, including statistical significance was set at P≤.01. Statistical ana- gender, age, and number of previous rotations. Rotation lysis was conducted using SAS version 9.3 software characteristics were collected, including rotation (SAS Institute, Inc.). specialty (general practice, medicine subspecialty, med- ical specialty, pediatrics, surgery, other), rotation type Results (required, elective), and length of rotation (4 or 6 weeks). Participant characteristics These demographic and rotation characteristics were Of 206 surveys sent to 41 unique PA students, 124 sur- chosen based on the authors’ hypothesis for potential as- veys were returned (60.2% response rate) by 33 students. sociations with PACRE scores and the availability of ac- Of the responses, 118 surveys from 32 students con- curate date. Each student ranked the following questions tained both evaluation and demographic data, and these on a 5-point Likert scale: 1) My program adequately pre- are the data source for this study. There were 28 stu- pared me for this rotation; and 2) This rotation prepared dents from the La Crosse program and 4 students were me for being a PA. from other PA programs. The 32 students completed be- Factor analysis was completed on the PACRE instru- tween 1 and 7 rotation evaluations. Student demograph- ment item scores. “To account for the clustering of ics are shown in Table 2. multiple ratings by students completing more than 1 rotation evaluation, we generated an adjusted correl- PACRE instrument validation ation matrix using generalized estimating equations. Factor analysis of the PACRE instrument showed a unidi- This adjusted correlation matrix was then used to per- mensional model for assessing PA clinical rotations (Table form factor analysis with orthogonal rotation. For a 1). Theeigenvaluefor the PACRE instrument’s one factor sensitivity analysis, we performed a factor analysis using was 9.63. Eigenvalues for all additional factors were < 1, an unadjusted correlation matrix and within-student which supports a one factor model (Fig. 1). Item factor combinations” [30]. Specifically, the sensitivity analysis loadings were all higher than 0.6, except for 2 items devel- involved conducting the factor analysis at lowest level oped for “promotion of self-directed learning.” These items of measurement (the student) and also higher levels of were removed from the remainder of the analysis and fu- measurement (group-averaged scores), and then com- ture iterations of the PACRE instrument. The internal paring these different levels of analysis to determine if consistency reliability was excellent (Cronbach α=0.95). they reveal similar or identical factor structures, which The item mean (SD) scores ranged from 4.08 (0.81) to 4.56 would then support reporting factor analysis of the (0.71). The mean overall PACRE score was 4.31 (0.57). higher, nested level of measurement. The number of factors to be retained was determined based on the Associations between PACRE scores and student or rotation eigenvalue criterion (factors with eigenvalues > 1). The characteristics final model was confirmed by reviewing the scree plot. PACRE scores were associated with the students’ gender. Items with factor loadings ≥0.60 were retained. Internal Men gave significantly higher PACRE scores than consistency reliability was calculated using the women (mean [SD], 4.7 [0.4] vs 4.3 [0.6]; P = .01). There Cronbach α,where α greater than 0.7 is considered ac- were no significant associations between rotation evalu- ceptable [31]. Additionally, for internal structure valid- ation scores and student age or PA program (tertiary ity determination an evaluation-to-item ratio should center vs health system). range from 5:1 to 10:1 [31]. The 18 items that make up PACRE scores were associated with the specialty of the PACRE instrument would require between 90 and the rotation. General practice rotations had the highest 180 completed instruments in order to be powered to PACRE scores (4.6 [0.5]), and surgical rotations had the complete the factor analysis. lowest (4.1 [0.5]; P = .006). There were no significant as- Categorical variables are presented as count (percent- sociations between rotation evaluation scores and rota- age) and continuous variables are presented as mean tion type or length. (SD). Associations between PACRE instrument scores Analysis indicated significant relationships between and categorical student demographic and clinical rota- PACRE scores and Likert-scaled responses to 1) My pro- tion characteristics were determined using the Wil- gram adequately prepared me for this rotation (Pearson coxon rank sum test (if 2 groups) or Kruskal-Wallace correlation coefficient, 0.32; P < .001) and 2) This rota- test (if more than 2 groups). Pearson correlation coeffi- tion prepared me for being a PA (Pearson correlation cients were calculated to explore the relationship be- coefficient, 0.57; P < .001). These represent a medium tween PACRE scores and continuous participant or correlation and a large correlation, respectively [32]. rotation characteristics (0.1–0.3, small correlation; 0.3– Regarding statistical power, for a binary variable with a 0.5, medium correlation; 0.5–1, large correlation) [32]. prevalence of 50%, the sample size of 118 students has Given multiple comparisons, the threshold for 80% power to detect a moderate effect size of 0.52 Meverden et al. BMC Medical Education (2018) 18:123 Page 4 of 7 Table 2 Associations Between PACRE Scores and Student or The newly developed PACRE questionnaire represents a Clinical Rotation Characteristics unidimensional model with strong internal consistency Characteristic Value PACRE P Value reliability. Student rotation evaluation scores were asso- (N = 118) Score ciated with the students’ gender and specialty of the ro- Student characteristics tation but not with whether it was required or elective. Gender .01 There was a positive correlation between the rotation evaluation scores and whether the student felt prepared Men 10 (8.5) 4.7 (0.4) for the rotation and whether they felt the rotation pre- Women 108 (91.5) 4.3 (0.6) pared them to be a PA. Age, y 24.1 (1.8) 4.3 (0.6) .07 This study adds to what is known about evaluations of No. of previous rotations 5.3 (3.0) 4.3 (0.6) .24 clinical teaching and rotations. Research on assessments Clinical rotation characteristics of clinical teaching in other educational settings has Specialty .006 been previously published. Factorial validation of the SFDP core components among medical students re- General practice 30 (25.4) 4.6 (0.5) e sulted in the SFDP26, a questionnaire that consists of 25 Medicine subspecialties 10 (8.5) 4.5 (0.4) items organized around these competencies and 1 item Medical specialties 24 (20.3) 4.2 (0.6) for overall effectiveness [24]. The SFDP26 has been ap- Pediatrics 9 (7.6) 4.2 (0.6) plied to resident education [33] and has been translated Surgery 29 (24.6) 4.1 (0.5) and validated in additional languages [23, 34, 35]. For Other 16 (13.6) 4.2 (0.6) example, a teaching evaluation form, which was based on the SFDP26 and our institution’s Faculty Resident Type .85 Electronic Evaluation System, was developed and tested Required 84 (71.2) 4.3 (0.6) among residents [20]. Research on clinical teaching of Elective 34 (28.8) 4.3 (0.6) PA students found that characteristics of an effective Location .46 and ineffective preceptor aligned with themes defined in Tertiary center 76 (64.4) 4.3 (0.6) the SFDP [36]. Two other studies used allied health stu- Health system 42 (35.6) 4.4 (0.6) dents (including some PA students) to evaluate effective clinical instruction and found similar results [37, 38]. Length of rotation, wk .39 The PACRE instrument described in this study is unique 4 115 (97.5) 4.3 (0.6) in that it was specifically designed for and tested in a PA 6 3 (2.5) 4.6 (0.4) student population and focused on the overall rotation My program adequately prepared 4.2 (0.8) 4.3 (0.6) <.001 experience, not just clinical teaching. me for this rotation The current study builds on prior work regarding stu- This rotation prepared me for 4.4 (0.7) 4.3 (0.6) <.001 dent and rotation factors associated with perceived being a PA qualityof the rotation.Wefound that rotation evalu- Abbreviations: PA physician assistant, PACRE physician assistant clinical ation scores were correlated with student gender. rotation evaluation Values are No. of responses (%) or mean (SD). Although we are unaware of studies exploring relation- Values are mean (SD). c ships between gender and rotation evaluation scores, Wilcoxon rank sum test (2 groups) or Kruskal-Wallace test (> 2 groups). Pearson correlation. previous work has demonstrated an association be- Cardiology [9], Endocrinology [1]. tween medical student gender and medical school per- Dermatology [5], Emergency Medicine [7], Neurology [2], Psychiatry [10]. formance [39, 40]. We found that clinical rotation Geriatrics [1], Hematology/Oncology [4], Infectious Disease [4], Interventional Radiology [1], Pain Management [2], Urology [2], Vascular Medicine [1], Women’s evaluation scores were associated with rotation spe- Health [1]. h cialty, feeling prepared for the rotation, and a better Pearson correlation coefficient, 0.32. Pearson correlation coefficient, 0.57. perception of the value of the rotation. Studies of med- ical students and residents have demonstrated that ro- standard deviations or higher based on a two sample of tation quality is related to rotation specialty [23, 41], t-test with equal variance. For continuous variables, a clinic schedule design [42], learning climate [43], re- sample of size 118 students has 80% power to detect a quirements for overnight call [44], quality of feedback correlation coefficient of 0.26 or higher between any two [44, 45], caseload [46], continuity [46], and level of fac- continuous variables, such as age versus PACRE score. ulty involvement [46]. In our study, associations be- tween rotation quality and rotation specialty suggest Discussion that differences between specialties exist and that fu- To our knowledge, this is the first study to report the ture studies could focus on elucidating these differ- validity of a clinical rotation assessment in PA education. ences. The finding that rotation evaluation scores Meverden et al. BMC Medical Education (2018) 18:123 Page 5 of 7 Fig. 1 Scree plot for the one factor model in the PACRE instrument. The scree plot shows the eigenvalues for the factors and supports the decision to retain the factors with an eigenvalue > 1 correlated with being prepared for the rotation is concord- suggest that reaction-level outcomes are commonly re- ant with the current trend in medical schools to offer ported [49]. Fifth, the utilization of 124 surveys from 33 transition courses for early clinical experiences [47]. students represents a relatively small number for factor The PACRE questionnaire has compelling validity evi- analysis. Sixth, the SFDP questionnaire was originally devel- dence. A common approach to validity in medical edu- oped for assessing only preceptors and our utilization of cation research includes content, internal structure, and the SFDP framework includes application of items to both relations to other variables evidence [48]. Content valid- preceptors and the program. However, most of the items ity for the PACRE questionnaire is based on published reflect students’ perceptions of their preceptors, the items assessments of clinical teaching [12, 14, 15, 20–29], it- that are applied to the program are applied in ways that are erative revision of instrument items, and pilot testing. true to the item’s original intent (e.g., “The rotation goals Internal structure validity is supported by a single factor were stated clearly”), and we note that an advantage is that to measure student perceptions’ of clinical rotations and ours is the first study to provide robust validity evidence for excellent internal consistency reliability. Relations to use of the SFDP framework for evaluation of students per- other variables validity evidence is demonstrated by as- ceptions of a PA program. Seventh, PACRE evaluation sociations between clinical rotation evaluation scores scores should be considered in the context of other out- and gender, rotation specialty, feeling prepared for the comes including faculty evaluations and knowledge assess- rotation, and viewing the rotation as valuable. Future ments (e.g. rotation examinations, observed structured study should determine if associations between PACRE clinical examinations) to fully evaluation a rotation. Finally, questionnaire scores and other rotation outcomes in- certain statistically significant score differences in this study cluding knowledge assessments exist. (e.g., general practice PACRE score = 4.6 versus surgical ro- tations PACRE score = 4.1; p = 0.006) may seem small; Limitations and strengths nonetheless, in many education settings the inflation and First, all students did rotations through 2 programs, which range restriction of assessment scores is very narrow, and could limit the generalizability of the findings. However, such magnitudes of difference could potentially separate analyses of published medical education studies indicate the best rotations from the rest. Strengths of this study in- that most are single institution studies [49]. Second, the clude a rigorous survey development process, use of a dedi- majority of responders in this study were female, which cated survey research program, robust statistical methods may limit generalizability. Third, while the response rate in to establish instrument validity, and high response rate. this study was excellent, there could be differences between those that did and did not complete the survey. Fourth, re- Conclusions sponses from the PACRE instrument are reaction outcomes This study has important implications for leaders of PA rather than the higher outcomes of learning, behavior, or clinical rotation experiences. First, we believe that this is results [50]. Yet, reviews of medical education research the first report of a valid method for assessing PA Meverden et al. BMC Medical Education (2018) 18:123 Page 6 of 7 clinical rotation experiences. Application of the PACRE [51 p.]. Available from: https://www.aamc.org/download/458082/data/2016_ complexities_of_supply_and_demand_projections.pdf questionnaire could inform rotation directors about 2. Green LV, Savin S, Lu Y. Primary care physician shortages could be ways to improve clinical experiences. Given that the de- eliminated through use of teams, nonphysicians, and electronic mand for PAs is rapidly increasing, the PACRE question- communication. Health Aff (Millwood). 2013;32(1):11–9. 3. Hofer AN, Abraham JM, Moscovice I. Expansion of coverage under the naire could provide a consistent technique for ensuring patient protection and affordable care act and primary care utilization. that rotations provide meaningful teaching and clinical Milbank Q. 2011;89(1):69–89. exposure. Second, the findings of this study suggest that 4. McDaniel MJ, Hildebrandt CA, Russell GB. Central application service for physician assistants ten-year data report, 2002 to 2011. J Physician Assist PA students must be adequately prepared to have a suc- Educ. 2016;27(1):17–23. cessful experience on their rotations. PA programs 5. Accreditation Review Commission on Education for the Physician Assistant. should consider offering transition courses like those of- Accredited Programs [Internet]. 2017 [cited 2017 Sept 18]. Available from: http://www.arc-pa.org/accreditation/accredited-programs/ fered in many medical schools to prepare their students 6. Erikson C, Hamann R, Levitan T, Pankow S, Stanley J, Whatley M; American for clinical experiences. Third, variability exists among Association of Colleges of Nursing; American Association of Colleges of specialties regarding perceived quality of rotations. PA Osteopathic Medicine; Association of American Medical Colleges; Physician Assistant Education Association. Recruiting and maintaining US clinical training programs should work to provide standard requirements sites: joint report of the 2013 multi-discipline clerkship/clinical training site for rotations such as a curriculum, evaluation standards, survey [Internet]. [cited 2017 Sept 18]; [24 p.]. Available from: https://members. competencies, and clinical exposure. Future research aamc.org/eweb/upload/13-225%20wc%20report%202%20update.pdf 7. Herrick A, Pearl J. Rotation shortages in physician assistant education should explore whether additional rotation characteris- [abstract]. J Am Acad Physician Assist. 2015;28(11):1. tics (e.g., didactic content, evaluation methods, call 8. Hudak NM, Enking PJ, Gorney C, Gonzalez-Colaso R. Tales from the trenches: schedules) and educational outcomes (e.g., learning, be- physician assistants’ perspectives about precepting students. J Physician Assist Educ. 2014;25(1):12–9. havior change) are associated with PACRE scores. 9. Porter S. Just-released clerkship study: growing shortage of clinical training sites challenges medical schools. Ann Fam Med. 2014;12(5):484–6. Abbreviations 10. Essary A, Berry-Lloyd C, Morgan C. A student-centered approach to physician PA: Physician assistant; PACRE: PA clinical rotation evaluation; SFDP: Stanford assistant clinical rotations. J Physician Assist Educ. 2003;14(2):78–83. Faculty Development Program 11. Winter RJ, Jones RJ. Clerkship evaluation by students: a standardized electronic mechanism. Acad Med. 1997;72(5):418. Funding 12. Bellini L, Shea JA, Asch DA. A new instrument for residency program The authors report no external funding source for this study. evaluation. J Gen Intern Med. 1997;12(11):707–10. 13. Seelig CB. Changes in residents’ attitudes in response to residency program Availability of data and materials modifications: a prospective study. South Med J. 1992;85(10):972–5. The datasets used and analyzed during the current study are available from 14. Seelig CB. Quantitating qualitative issues in residency training: development the corresponding author on reasonable request. and testing of a scaled program evaluation questionnaire. J Gen Intern Med. 1993;8(11):610–3. Authors’ contributions 15. Seelig CB, DuPre CT, Adelman HM. Development and validation of a scaled RM, JS, TB, CW contributed to the design, data collection, analysis, and drafting questionnaire for evaluation of residency programs. South Med J. 1995; of the manuscript. SM, CS, JM contributed to the data collection, analysis, and 88(7):745–50. drafting of the manuscript. All authors read and approved the final manuscript. 16. Kirton OC, Antonetti M, Morejon O, Dobkin E, Angelica MD, Reilly PJ, et al. Measuring service-specific performance and educational value within a Ethics approval and consent to participate general surgery residency: the power of a prospective, anonymous, web- This study was deemed exempt by the Mayo Clinic institutional review board. based rotation evaluation system in the optimization of resident Given that the research was conducted in an established educational setting satisfaction. Surgery. 2001;130(2):289–95. involving normal educational practices, the Mayo Clinic institutional review 17. Davisson L, Nuss M, Cottrell S. Women’s health curriculum for internal board did not require consent from the participants. medicine residents: development, implementation, and evaluation. J Grad Med Educ. 2010;2(3):398–403. 18. Akl EA, Maroun N, Klocke RA, Schunemann HJ. A survey of internal Competing interests medicine residents and faculty about the duration of attendings’ inpatient The authors declare they have no competing interest. rotations. J Gen Intern Med. 2004;19(11):1133–9. 19. Tanaka P, Yanez D, Lemmens H, Djurdjulov A, Scotto L, Borg L, et al. Impact Publisher’sNote of an innovative classroom-based lecture series on residents’ evaluations of Springer Nature remains neutral with regard to jurisdictional claims in an anesthesiology rotation. Anesthesiol Res Pract. 2016;2016:8543809. published maps and institutional affiliations. 20. Beckman TJ, Lee MC, Rohren CH, Pankratz VS. Evaluating an instrument for the peer review of inpatient teaching. Med Teach. 2003;25(2):131–5. Author details 21. Dolmans DH, Wolfhagen IH, Essed GG, Scherpbier AJ, van der Vleuten CP. Mayo Clinic Gonda Vascular Center, Mayo Clinic, Rochester, MN, USA. The impacts of supervision, patient mix, and numbers of students on the Division of General Internal Medicine, Mayo Clinic, 200 First St. SW, effectiveness of clinical rotations. Acad Med. 2002;77(4):332–5. Rochester, MN 55905, USA. Division of Biomedical Statistics and Informatics, 22. Hunter AJ, Desai SS, Harrison RA, Chan BK. Medical student evaluation of Mayo Clinic, Rochester, MN, USA. the quality of hospitalist and nonhospitalist teaching faculty on inpatient medicine rotations. Acad Med. 2004;79(1):78–82. Received: 2 January 2018 Accepted: 25 May 2018 23. Kihlberg P, Perzon M, Gedeborg R, Blomqvist P, Johansson J. Uniform evaluation of clinical teaching: an instrument for specific feedback and cross comparison between departments. Högre Utbildning. 2011;1(2): References 139–50. 1. Dall T, West T, Chakrabarti R, Iacobucci W; Association of American Medical 24. Litzelman DK, Stratos GA, Marriott DJ, Skeff KM. Factorial validation of a Colleges. The complexities of physician supply and demand: 2016 update: widely disseminated educational framework for evaluating clinical teachers. projections from 2014 to 2025 [Internet]. 2016 Apr 5 [cited 2017 Sept 18]; Acad Med. 1998;73(6):688–95. Meverden et al. BMC Medical Education (2018) 18:123 Page 7 of 7 25. Owolabi MO. Development and psychometric characteristics of a new domain of the Stanford faculty development program instrument. J Contin Educ Heal Prof. 2014;34(1):13–24. 26. Skeff KM, Stratos GA, Berman J, Bergen MR. Improving clinical teaching: evaluation of a national dissemination program. Arch Intern Med. 1992; 152(6):1156–61. 27. Sutkin G, Wagner E, Harris I, Schiffer R. What makes a good clinical teacher in medicine? A review of the literature. Acad Med. 2008;83(5):452–66. 28. Valdivia TD, Hartquist BA. An ACCESS-based academic attending physician and resident rotation evaluation database. Proc Annu Symp Comput Appl Med Care. 1994;1000. 29. Skeff KM. Enhancing teaching effectiveness and vitality in the ambulatory setting. J Gen Intern Med. 1988;3(2 Suppl):S26–33. 30. Post JA, Wittich CM, Thomas KG, Dupras DM, Halvorsen AJ, Mandrekar JN, et al. Rating the quality of entrustable professional activities: content validation and associations with the clinical context. J Gen Intern Med. 2016; 31(5):518–23. Epub 2016 Feb 22 31. RF DV. Scale development: theory and applications. Newbury Park (CA): Sage. p. c1991. 32. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale (NJ): L. In: Erlbaum associates; 1988. 33. Litzelman DK, Westmoreland GR, Skeff KM, Stratos GA. Factorial validation of an educational framework using residents’ evaluations of clinician- educators. Acad Med. 1999;74(10 Suppl):S25–7. 34. Iblher P, Zupanic M, Hartel C, Heinze H, Schmucker P, Fischer MR. The questionnaire “SFDP26-German”: a reliable tool for evaluation of clinical teaching? GMS Z Med Ausbild 2011;28(2):Doc30. 35. Wong JG, Agisheva K. Developing teaching skills for medical educators in Russia: a cross-cultural faculty development project. Med Educ. 2007;41(3): 318–24. 36. Zayas TT. Qualities of effective preceptors of physician assistant students. J Phycian Assist Educ. 1999;10(1):7–11. 37. Jarski RW, Kulig K, Olson RE. Allied health perceptions of effective clinical instruction. J Allied Health. 1989;18(5):469–78. 38. Rogers JL, Lautar CJ, Dunn LR. Allied health students’ perceptions of effective clinical instruction. Health Care Manag (Frederick). 2010;29(1):63–7. 39. Bienstock JL, Martin S, Tzou W, Fox HE. Medical students’ gender is a predictor of success in the obstetrics and gynecology basic clerkship. Teach Learn Med. 2002;14(4):240–3. 40. Haist SA, Wilson JF, Elam CL, Blue AV, Fosson SE. The effect of gender and age on medical school performance: an important interaction. Adv Health Sci Educ Theory Pract. 2000;5(3):197–205. 41. Lewisson N, Hellgren L, Johansson J. Quality improvement in clinical teaching through student evaluations of rotations and feedback to departments. Med Teach. 2013;35(10):820–5. 42. Wieland ML, Halvorsen AJ, Chaudhry R, Reed DA, McDonald FS, Thomas KG. An evaluation of internal medicine residency continuity clinic redesign to a 50/50 outpatient-inpatient model. J Gen Intern Med. 2013;28(8):1014–9. 43. Dyrbye LN, Thomas MR, Harper W, Massie FS Jr, Power DV, Eacker A, et al. The learning environment and medical student burnout: a multicentre study. Med Educ. 2009;43(3):274–82. 44. Remmen R, Denekens J, Scherpbier A, Hermann I, van der Vleuten C, Royen PV, et al. An evaluation study of the didactic quality of clerkships. Med Educ. 2000;34(6):460–4. 45. Torre DM, Sebastian JL, Simpson DE. Learning activities and high-quality teaching: perceptions of third-year IM clerkship students. Acad Med. 2003; 78(8):812–4. 46. Schneider JR, Coyle JJ, Ryan ER, Bell RH Jr, DaRosa DA. Implementation and evaluation of a new surgical residency model. J Am Coll Surg. 2007;205(3): 393–404. 47. Poncelet A, O’Brien B. Preparing medical students for clerkships: a descriptive analysis of transition courses. Acad Med. 2008;83(5):444–51. 48. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166.e7–16. 49. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:1002–9. 50. Kirkpatrick D. Revisiting Kirkpatrick's four-level model. Training and. Development. 1996;50:54–9. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png BMC Medical Education Springer Journals

Validation of a clinical rotation evaluation for physician assistant students

Free
7 pages

Loading next page...
 
/lp/springer_journal/validation-of-a-clinical-rotation-evaluation-for-physician-assistant-qOF9oBEEUd
Publisher
Springer Journals
Copyright
Copyright © 2018 by The Author(s).
Subject
Education; Medical Education; Theory of Medicine/Bioethics
eISSN
1472-6920
D.O.I.
10.1186/s12909-018-1242-y
Publisher site
See Article on Publisher Site

Abstract

Background: We conducted a prospective validation study to develop a physician assistant (PA) clinical rotation evaluation (PACRE) instrument. The specific aims of this study were to 1) develop a tool to evaluate PA clinical rotations, and 2) explore associations between validated rotation evaluation scores and characteristics of the students and rotations. Methods: The PACRE was administered to rotating PA students at our institution in 2016. Factor analysis, internal consistency reliability, and associations between PACRE scores and student or rotation characteristics were determined. Results: Of 206 PACRE instruments sent, 124 were returned (60.2% response). Factor analysis supported a unidimensional model with a mean (SD) score of 4.31 (0.57) on a 5-point scale. Internal consistency reliability was excellent (Cronbach α=0.95). PACRE scores were associated with students’ gender (P = .01) and rotation specialty (P = .006) and correlated with students’ perception of being prepared (r = 0.32; P < .001) and value of the rotation (r =0.57; P < .001). Conclusions: This is the first validated instrument to evaluate PA rotation experiences. Application of the PACRE questionnaire could inform rotation directors about ways to improve clinical experiences. The findings of this study suggest that PA students must be adequately prepared to have a successful experience on their rotations. PA programs should consider offering transition courses like those offered in many medical schools to prepare their students for clinical experiences. Future research should explore whether additional rotation characteristics and educational outcomes are associated with PACRE scores. Keywords: Assessment, Clinical rotation, Physician assistant education, Validation study Background Data are sparse regarding the evaluation of training Physician assistants (PAs) are vital to all aspects of programs and individual clinical rotations in other edu- health care delivery. The number of PA training pro- cational settings. In medical school, students have evalu- grams continues to increase to meet the demand for ac- ated clerkships using electronic Likert-scaled checklists cess to health care [1–3]. In the past decade, the number [11]. In graduate medical education, residents have used of accredited PA programs and applicants to these pro- validated questionnaires to evaluate their programs with grams has increased dramatically [4, 5]. An integral part respect to rotation workload, faculty/learning environ- of these programs are clinical rotations, which are lim- ment, and stress [12–15]. Other studies have examined ited by competition and shortages [6–9]. The educa- resident assessments of programs and rotations in sur- tional quality of these clinical rotations can vary [10]. gery [16], internal medicine [17, 18], and anesthesiology Maintaining standards in clinical rotations, including [19]. Although PAs are trained in the medical model, validated assessment of performance, is a prerequisite to differences in clinical rotation length, content, supervi- ensuring a high-quality PA workforce. sion, and logistics make the use of existing clinical rota- tion evaluations less relevant to PA training settings. We are unaware of prior research on validated measures of * Correspondence: wittich.christopher@mayo.edu PA clinical rotations. Division of General Internal Medicine, Mayo Clinic, 200 First St. SW, Rochester, MN 55905, USA Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Meverden et al. BMC Medical Education (2018) 18:123 Page 2 of 7 To fill this gap, we conducted a prospective validation PACRE instrument development study to develop a PA clinical rotation evaluation A PACRE questionnaire was developed on the basis of (PACRE) instrument. The specific aims of this study existing literature [12, 14, 15, 20–28]. Items were devel- were to 1) develop a tool to evaluate PA clinical oped for each of the Stanford Faculty Development rotations, and 2) once validated, explore associations be- Program (SFDP) for Clinical Teaching categories: learn- tween PACRE rotation evaluation scores and character- ing climate, control of session, communication of goals, istics of the students and rotations. The purpose of the promotion of understanding and retention, evaluation, PACRE instrument was to determine and document a feedback, and promotion of self-directed learning [24, 29]. student’s perceptions of a rotation based on research on Two additional categories—rotation logistics and a rota- components of effective clinical teaching. We hypothe- tion global assessment—were included. After iterative re- sized that such an instrument would have strong in- vision, 2 items were selected for each of the 9 domains, ternal structure validity and scores would be associated for a total of 18 items in the PACRE instrument (Table 1). with rotation or student demographic variables. Responses were based on a 5-point Likert scale (1, strongly disagree, to 5, strongly agree). The final PACRE Methods instrument was pilot tested on 5 former PA students and Participants and clinical sites 5 current PA colleagues, which led to minor rewording of This prospective validation study involved PA students some items. who completed a clinical rotation in 2016 at1) the Mayo Clinic School of Health Sciences/Gundersen Medical Data collection and analysis Foundation/University of Wisconsin – La Crosse PA The PACRE was sent via an emailed link to each PA stu- Program or 2) the Mayo Clinic or the Mayo Clinic dent at the completion of the clinical rotation. Our sur- Health System from other PA programs. The study was vey research center managed the data collection using deemed exempt by the Mayo Clinic institutional review Qualtrics software (Qualtrics LLC). A reminder email board (Identification number: 15–006040). was sent 1 week after the completion of the rotation. Table 1 PACRE Instrument Educational Domains, Items, Item Loadings, and Mean Scores Educational Domain PACRE Item Item Mean (SD) Loading PACRE Score Learning climate The preceptor (s) created an environment that was conducive to learning. 0.7958 4.40 (0.75) The preceptor (s) was/were enthusiastic. 0.6585 4.56 (0.71) Control of session The preceptor (s) balanced time between patient care and teaching. 0.7751 4.35 (0.85) The preceptor (s) utilized my time effectively. 0.7515 4.24 (0.81) Communication of goals The rotation goals were stated clearly. 0.7937 4.20 (0.82) The rotation goals were appropriate for my educational needs. 0.7108 4.38 (0.69) Promotion of understanding The educational content was clearly communicated. 0.7536 4.33 (0.70) and retention The preceptor (s) helped to facilitate my understanding and retention 0.7770 4.47 (0.65) of information. Evaluation My performance was assessed by the preceptor (s). 0.8028 4.26 (0.72) I was evaluated on what I learned. 0.7788 4.08 (0.81) Feedback I received feedback on my performance. 0.8524 4.22 (0.75) The preceptor (s) communicated constructive assessments of my abilities. 0.8188 4.22 (0.79) Promotion of self-directed learning I had access to educational resources. 0.2183 4.68 (0.49) I was encouraged to learn on my own. 0.5270 4.54 (0.55) Rotation logistics I was oriented to information regarding rotation logistics. 0.7360 4.16 (0.75) My overall experience was organized. 0.7626 4.37 (0.68) Global assessment I would recommend this rotation. 0.7463 4.36 (0.82) I would recommend this site/work area as a place of employment. 0.6587 4.41 (0.75) Overall (16 items) 4.31 (0.57) Abbreviation: PACRE physician assistant clinical rotation evaluation Items from this educational domain loaded ambiguously and were excluded. Cronbach α (internal consistency reliability) was 0.95 for the 16 retained items. The eigenvalue for factor 1 was 9.63. Eigenvalues for all additional factors were < 1, which supports a one factor model for the PACRE instrument. Meverden et al. BMC Medical Education (2018) 18:123 Page 3 of 7 Demographic characteristics were collected, including statistical significance was set at P≤.01. Statistical ana- gender, age, and number of previous rotations. Rotation lysis was conducted using SAS version 9.3 software characteristics were collected, including rotation (SAS Institute, Inc.). specialty (general practice, medicine subspecialty, med- ical specialty, pediatrics, surgery, other), rotation type Results (required, elective), and length of rotation (4 or 6 weeks). Participant characteristics These demographic and rotation characteristics were Of 206 surveys sent to 41 unique PA students, 124 sur- chosen based on the authors’ hypothesis for potential as- veys were returned (60.2% response rate) by 33 students. sociations with PACRE scores and the availability of ac- Of the responses, 118 surveys from 32 students con- curate date. Each student ranked the following questions tained both evaluation and demographic data, and these on a 5-point Likert scale: 1) My program adequately pre- are the data source for this study. There were 28 stu- pared me for this rotation; and 2) This rotation prepared dents from the La Crosse program and 4 students were me for being a PA. from other PA programs. The 32 students completed be- Factor analysis was completed on the PACRE instru- tween 1 and 7 rotation evaluations. Student demograph- ment item scores. “To account for the clustering of ics are shown in Table 2. multiple ratings by students completing more than 1 rotation evaluation, we generated an adjusted correl- PACRE instrument validation ation matrix using generalized estimating equations. Factor analysis of the PACRE instrument showed a unidi- This adjusted correlation matrix was then used to per- mensional model for assessing PA clinical rotations (Table form factor analysis with orthogonal rotation. For a 1). Theeigenvaluefor the PACRE instrument’s one factor sensitivity analysis, we performed a factor analysis using was 9.63. Eigenvalues for all additional factors were < 1, an unadjusted correlation matrix and within-student which supports a one factor model (Fig. 1). Item factor combinations” [30]. Specifically, the sensitivity analysis loadings were all higher than 0.6, except for 2 items devel- involved conducting the factor analysis at lowest level oped for “promotion of self-directed learning.” These items of measurement (the student) and also higher levels of were removed from the remainder of the analysis and fu- measurement (group-averaged scores), and then com- ture iterations of the PACRE instrument. The internal paring these different levels of analysis to determine if consistency reliability was excellent (Cronbach α=0.95). they reveal similar or identical factor structures, which The item mean (SD) scores ranged from 4.08 (0.81) to 4.56 would then support reporting factor analysis of the (0.71). The mean overall PACRE score was 4.31 (0.57). higher, nested level of measurement. The number of factors to be retained was determined based on the Associations between PACRE scores and student or rotation eigenvalue criterion (factors with eigenvalues > 1). The characteristics final model was confirmed by reviewing the scree plot. PACRE scores were associated with the students’ gender. Items with factor loadings ≥0.60 were retained. Internal Men gave significantly higher PACRE scores than consistency reliability was calculated using the women (mean [SD], 4.7 [0.4] vs 4.3 [0.6]; P = .01). There Cronbach α,where α greater than 0.7 is considered ac- were no significant associations between rotation evalu- ceptable [31]. Additionally, for internal structure valid- ation scores and student age or PA program (tertiary ity determination an evaluation-to-item ratio should center vs health system). range from 5:1 to 10:1 [31]. The 18 items that make up PACRE scores were associated with the specialty of the PACRE instrument would require between 90 and the rotation. General practice rotations had the highest 180 completed instruments in order to be powered to PACRE scores (4.6 [0.5]), and surgical rotations had the complete the factor analysis. lowest (4.1 [0.5]; P = .006). There were no significant as- Categorical variables are presented as count (percent- sociations between rotation evaluation scores and rota- age) and continuous variables are presented as mean tion type or length. (SD). Associations between PACRE instrument scores Analysis indicated significant relationships between and categorical student demographic and clinical rota- PACRE scores and Likert-scaled responses to 1) My pro- tion characteristics were determined using the Wil- gram adequately prepared me for this rotation (Pearson coxon rank sum test (if 2 groups) or Kruskal-Wallace correlation coefficient, 0.32; P < .001) and 2) This rota- test (if more than 2 groups). Pearson correlation coeffi- tion prepared me for being a PA (Pearson correlation cients were calculated to explore the relationship be- coefficient, 0.57; P < .001). These represent a medium tween PACRE scores and continuous participant or correlation and a large correlation, respectively [32]. rotation characteristics (0.1–0.3, small correlation; 0.3– Regarding statistical power, for a binary variable with a 0.5, medium correlation; 0.5–1, large correlation) [32]. prevalence of 50%, the sample size of 118 students has Given multiple comparisons, the threshold for 80% power to detect a moderate effect size of 0.52 Meverden et al. BMC Medical Education (2018) 18:123 Page 4 of 7 Table 2 Associations Between PACRE Scores and Student or The newly developed PACRE questionnaire represents a Clinical Rotation Characteristics unidimensional model with strong internal consistency Characteristic Value PACRE P Value reliability. Student rotation evaluation scores were asso- (N = 118) Score ciated with the students’ gender and specialty of the ro- Student characteristics tation but not with whether it was required or elective. Gender .01 There was a positive correlation between the rotation evaluation scores and whether the student felt prepared Men 10 (8.5) 4.7 (0.4) for the rotation and whether they felt the rotation pre- Women 108 (91.5) 4.3 (0.6) pared them to be a PA. Age, y 24.1 (1.8) 4.3 (0.6) .07 This study adds to what is known about evaluations of No. of previous rotations 5.3 (3.0) 4.3 (0.6) .24 clinical teaching and rotations. Research on assessments Clinical rotation characteristics of clinical teaching in other educational settings has Specialty .006 been previously published. Factorial validation of the SFDP core components among medical students re- General practice 30 (25.4) 4.6 (0.5) e sulted in the SFDP26, a questionnaire that consists of 25 Medicine subspecialties 10 (8.5) 4.5 (0.4) items organized around these competencies and 1 item Medical specialties 24 (20.3) 4.2 (0.6) for overall effectiveness [24]. The SFDP26 has been ap- Pediatrics 9 (7.6) 4.2 (0.6) plied to resident education [33] and has been translated Surgery 29 (24.6) 4.1 (0.5) and validated in additional languages [23, 34, 35]. For Other 16 (13.6) 4.2 (0.6) example, a teaching evaluation form, which was based on the SFDP26 and our institution’s Faculty Resident Type .85 Electronic Evaluation System, was developed and tested Required 84 (71.2) 4.3 (0.6) among residents [20]. Research on clinical teaching of Elective 34 (28.8) 4.3 (0.6) PA students found that characteristics of an effective Location .46 and ineffective preceptor aligned with themes defined in Tertiary center 76 (64.4) 4.3 (0.6) the SFDP [36]. Two other studies used allied health stu- Health system 42 (35.6) 4.4 (0.6) dents (including some PA students) to evaluate effective clinical instruction and found similar results [37, 38]. Length of rotation, wk .39 The PACRE instrument described in this study is unique 4 115 (97.5) 4.3 (0.6) in that it was specifically designed for and tested in a PA 6 3 (2.5) 4.6 (0.4) student population and focused on the overall rotation My program adequately prepared 4.2 (0.8) 4.3 (0.6) <.001 experience, not just clinical teaching. me for this rotation The current study builds on prior work regarding stu- This rotation prepared me for 4.4 (0.7) 4.3 (0.6) <.001 dent and rotation factors associated with perceived being a PA qualityof the rotation.Wefound that rotation evalu- Abbreviations: PA physician assistant, PACRE physician assistant clinical ation scores were correlated with student gender. rotation evaluation Values are No. of responses (%) or mean (SD). Although we are unaware of studies exploring relation- Values are mean (SD). c ships between gender and rotation evaluation scores, Wilcoxon rank sum test (2 groups) or Kruskal-Wallace test (> 2 groups). Pearson correlation. previous work has demonstrated an association be- Cardiology [9], Endocrinology [1]. tween medical student gender and medical school per- Dermatology [5], Emergency Medicine [7], Neurology [2], Psychiatry [10]. formance [39, 40]. We found that clinical rotation Geriatrics [1], Hematology/Oncology [4], Infectious Disease [4], Interventional Radiology [1], Pain Management [2], Urology [2], Vascular Medicine [1], Women’s evaluation scores were associated with rotation spe- Health [1]. h cialty, feeling prepared for the rotation, and a better Pearson correlation coefficient, 0.32. Pearson correlation coefficient, 0.57. perception of the value of the rotation. Studies of med- ical students and residents have demonstrated that ro- standard deviations or higher based on a two sample of tation quality is related to rotation specialty [23, 41], t-test with equal variance. For continuous variables, a clinic schedule design [42], learning climate [43], re- sample of size 118 students has 80% power to detect a quirements for overnight call [44], quality of feedback correlation coefficient of 0.26 or higher between any two [44, 45], caseload [46], continuity [46], and level of fac- continuous variables, such as age versus PACRE score. ulty involvement [46]. In our study, associations be- tween rotation quality and rotation specialty suggest Discussion that differences between specialties exist and that fu- To our knowledge, this is the first study to report the ture studies could focus on elucidating these differ- validity of a clinical rotation assessment in PA education. ences. The finding that rotation evaluation scores Meverden et al. BMC Medical Education (2018) 18:123 Page 5 of 7 Fig. 1 Scree plot for the one factor model in the PACRE instrument. The scree plot shows the eigenvalues for the factors and supports the decision to retain the factors with an eigenvalue > 1 correlated with being prepared for the rotation is concord- suggest that reaction-level outcomes are commonly re- ant with the current trend in medical schools to offer ported [49]. Fifth, the utilization of 124 surveys from 33 transition courses for early clinical experiences [47]. students represents a relatively small number for factor The PACRE questionnaire has compelling validity evi- analysis. Sixth, the SFDP questionnaire was originally devel- dence. A common approach to validity in medical edu- oped for assessing only preceptors and our utilization of cation research includes content, internal structure, and the SFDP framework includes application of items to both relations to other variables evidence [48]. Content valid- preceptors and the program. However, most of the items ity for the PACRE questionnaire is based on published reflect students’ perceptions of their preceptors, the items assessments of clinical teaching [12, 14, 15, 20–29], it- that are applied to the program are applied in ways that are erative revision of instrument items, and pilot testing. true to the item’s original intent (e.g., “The rotation goals Internal structure validity is supported by a single factor were stated clearly”), and we note that an advantage is that to measure student perceptions’ of clinical rotations and ours is the first study to provide robust validity evidence for excellent internal consistency reliability. Relations to use of the SFDP framework for evaluation of students per- other variables validity evidence is demonstrated by as- ceptions of a PA program. Seventh, PACRE evaluation sociations between clinical rotation evaluation scores scores should be considered in the context of other out- and gender, rotation specialty, feeling prepared for the comes including faculty evaluations and knowledge assess- rotation, and viewing the rotation as valuable. Future ments (e.g. rotation examinations, observed structured study should determine if associations between PACRE clinical examinations) to fully evaluation a rotation. Finally, questionnaire scores and other rotation outcomes in- certain statistically significant score differences in this study cluding knowledge assessments exist. (e.g., general practice PACRE score = 4.6 versus surgical ro- tations PACRE score = 4.1; p = 0.006) may seem small; Limitations and strengths nonetheless, in many education settings the inflation and First, all students did rotations through 2 programs, which range restriction of assessment scores is very narrow, and could limit the generalizability of the findings. However, such magnitudes of difference could potentially separate analyses of published medical education studies indicate the best rotations from the rest. Strengths of this study in- that most are single institution studies [49]. Second, the clude a rigorous survey development process, use of a dedi- majority of responders in this study were female, which cated survey research program, robust statistical methods may limit generalizability. Third, while the response rate in to establish instrument validity, and high response rate. this study was excellent, there could be differences between those that did and did not complete the survey. Fourth, re- Conclusions sponses from the PACRE instrument are reaction outcomes This study has important implications for leaders of PA rather than the higher outcomes of learning, behavior, or clinical rotation experiences. First, we believe that this is results [50]. Yet, reviews of medical education research the first report of a valid method for assessing PA Meverden et al. BMC Medical Education (2018) 18:123 Page 6 of 7 clinical rotation experiences. Application of the PACRE [51 p.]. Available from: https://www.aamc.org/download/458082/data/2016_ complexities_of_supply_and_demand_projections.pdf questionnaire could inform rotation directors about 2. Green LV, Savin S, Lu Y. Primary care physician shortages could be ways to improve clinical experiences. Given that the de- eliminated through use of teams, nonphysicians, and electronic mand for PAs is rapidly increasing, the PACRE question- communication. Health Aff (Millwood). 2013;32(1):11–9. 3. Hofer AN, Abraham JM, Moscovice I. Expansion of coverage under the naire could provide a consistent technique for ensuring patient protection and affordable care act and primary care utilization. that rotations provide meaningful teaching and clinical Milbank Q. 2011;89(1):69–89. exposure. Second, the findings of this study suggest that 4. McDaniel MJ, Hildebrandt CA, Russell GB. Central application service for physician assistants ten-year data report, 2002 to 2011. J Physician Assist PA students must be adequately prepared to have a suc- Educ. 2016;27(1):17–23. cessful experience on their rotations. PA programs 5. Accreditation Review Commission on Education for the Physician Assistant. should consider offering transition courses like those of- Accredited Programs [Internet]. 2017 [cited 2017 Sept 18]. Available from: http://www.arc-pa.org/accreditation/accredited-programs/ fered in many medical schools to prepare their students 6. Erikson C, Hamann R, Levitan T, Pankow S, Stanley J, Whatley M; American for clinical experiences. Third, variability exists among Association of Colleges of Nursing; American Association of Colleges of specialties regarding perceived quality of rotations. PA Osteopathic Medicine; Association of American Medical Colleges; Physician Assistant Education Association. Recruiting and maintaining US clinical training programs should work to provide standard requirements sites: joint report of the 2013 multi-discipline clerkship/clinical training site for rotations such as a curriculum, evaluation standards, survey [Internet]. [cited 2017 Sept 18]; [24 p.]. Available from: https://members. competencies, and clinical exposure. Future research aamc.org/eweb/upload/13-225%20wc%20report%202%20update.pdf 7. Herrick A, Pearl J. Rotation shortages in physician assistant education should explore whether additional rotation characteris- [abstract]. J Am Acad Physician Assist. 2015;28(11):1. tics (e.g., didactic content, evaluation methods, call 8. Hudak NM, Enking PJ, Gorney C, Gonzalez-Colaso R. Tales from the trenches: schedules) and educational outcomes (e.g., learning, be- physician assistants’ perspectives about precepting students. J Physician Assist Educ. 2014;25(1):12–9. havior change) are associated with PACRE scores. 9. Porter S. Just-released clerkship study: growing shortage of clinical training sites challenges medical schools. Ann Fam Med. 2014;12(5):484–6. Abbreviations 10. Essary A, Berry-Lloyd C, Morgan C. A student-centered approach to physician PA: Physician assistant; PACRE: PA clinical rotation evaluation; SFDP: Stanford assistant clinical rotations. J Physician Assist Educ. 2003;14(2):78–83. Faculty Development Program 11. Winter RJ, Jones RJ. Clerkship evaluation by students: a standardized electronic mechanism. Acad Med. 1997;72(5):418. Funding 12. Bellini L, Shea JA, Asch DA. A new instrument for residency program The authors report no external funding source for this study. evaluation. J Gen Intern Med. 1997;12(11):707–10. 13. Seelig CB. Changes in residents’ attitudes in response to residency program Availability of data and materials modifications: a prospective study. South Med J. 1992;85(10):972–5. The datasets used and analyzed during the current study are available from 14. Seelig CB. Quantitating qualitative issues in residency training: development the corresponding author on reasonable request. and testing of a scaled program evaluation questionnaire. J Gen Intern Med. 1993;8(11):610–3. Authors’ contributions 15. Seelig CB, DuPre CT, Adelman HM. Development and validation of a scaled RM, JS, TB, CW contributed to the design, data collection, analysis, and drafting questionnaire for evaluation of residency programs. South Med J. 1995; of the manuscript. SM, CS, JM contributed to the data collection, analysis, and 88(7):745–50. drafting of the manuscript. All authors read and approved the final manuscript. 16. Kirton OC, Antonetti M, Morejon O, Dobkin E, Angelica MD, Reilly PJ, et al. Measuring service-specific performance and educational value within a Ethics approval and consent to participate general surgery residency: the power of a prospective, anonymous, web- This study was deemed exempt by the Mayo Clinic institutional review board. based rotation evaluation system in the optimization of resident Given that the research was conducted in an established educational setting satisfaction. Surgery. 2001;130(2):289–95. involving normal educational practices, the Mayo Clinic institutional review 17. Davisson L, Nuss M, Cottrell S. Women’s health curriculum for internal board did not require consent from the participants. medicine residents: development, implementation, and evaluation. J Grad Med Educ. 2010;2(3):398–403. 18. Akl EA, Maroun N, Klocke RA, Schunemann HJ. A survey of internal Competing interests medicine residents and faculty about the duration of attendings’ inpatient The authors declare they have no competing interest. rotations. J Gen Intern Med. 2004;19(11):1133–9. 19. Tanaka P, Yanez D, Lemmens H, Djurdjulov A, Scotto L, Borg L, et al. Impact Publisher’sNote of an innovative classroom-based lecture series on residents’ evaluations of Springer Nature remains neutral with regard to jurisdictional claims in an anesthesiology rotation. Anesthesiol Res Pract. 2016;2016:8543809. published maps and institutional affiliations. 20. Beckman TJ, Lee MC, Rohren CH, Pankratz VS. Evaluating an instrument for the peer review of inpatient teaching. Med Teach. 2003;25(2):131–5. Author details 21. Dolmans DH, Wolfhagen IH, Essed GG, Scherpbier AJ, van der Vleuten CP. Mayo Clinic Gonda Vascular Center, Mayo Clinic, Rochester, MN, USA. The impacts of supervision, patient mix, and numbers of students on the Division of General Internal Medicine, Mayo Clinic, 200 First St. SW, effectiveness of clinical rotations. Acad Med. 2002;77(4):332–5. Rochester, MN 55905, USA. Division of Biomedical Statistics and Informatics, 22. Hunter AJ, Desai SS, Harrison RA, Chan BK. Medical student evaluation of Mayo Clinic, Rochester, MN, USA. the quality of hospitalist and nonhospitalist teaching faculty on inpatient medicine rotations. Acad Med. 2004;79(1):78–82. Received: 2 January 2018 Accepted: 25 May 2018 23. Kihlberg P, Perzon M, Gedeborg R, Blomqvist P, Johansson J. Uniform evaluation of clinical teaching: an instrument for specific feedback and cross comparison between departments. Högre Utbildning. 2011;1(2): References 139–50. 1. Dall T, West T, Chakrabarti R, Iacobucci W; Association of American Medical 24. Litzelman DK, Stratos GA, Marriott DJ, Skeff KM. Factorial validation of a Colleges. The complexities of physician supply and demand: 2016 update: widely disseminated educational framework for evaluating clinical teachers. projections from 2014 to 2025 [Internet]. 2016 Apr 5 [cited 2017 Sept 18]; Acad Med. 1998;73(6):688–95. Meverden et al. BMC Medical Education (2018) 18:123 Page 7 of 7 25. Owolabi MO. Development and psychometric characteristics of a new domain of the Stanford faculty development program instrument. J Contin Educ Heal Prof. 2014;34(1):13–24. 26. Skeff KM, Stratos GA, Berman J, Bergen MR. Improving clinical teaching: evaluation of a national dissemination program. Arch Intern Med. 1992; 152(6):1156–61. 27. Sutkin G, Wagner E, Harris I, Schiffer R. What makes a good clinical teacher in medicine? A review of the literature. Acad Med. 2008;83(5):452–66. 28. Valdivia TD, Hartquist BA. An ACCESS-based academic attending physician and resident rotation evaluation database. Proc Annu Symp Comput Appl Med Care. 1994;1000. 29. Skeff KM. Enhancing teaching effectiveness and vitality in the ambulatory setting. J Gen Intern Med. 1988;3(2 Suppl):S26–33. 30. Post JA, Wittich CM, Thomas KG, Dupras DM, Halvorsen AJ, Mandrekar JN, et al. Rating the quality of entrustable professional activities: content validation and associations with the clinical context. J Gen Intern Med. 2016; 31(5):518–23. Epub 2016 Feb 22 31. RF DV. Scale development: theory and applications. Newbury Park (CA): Sage. p. c1991. 32. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale (NJ): L. In: Erlbaum associates; 1988. 33. Litzelman DK, Westmoreland GR, Skeff KM, Stratos GA. Factorial validation of an educational framework using residents’ evaluations of clinician- educators. Acad Med. 1999;74(10 Suppl):S25–7. 34. Iblher P, Zupanic M, Hartel C, Heinze H, Schmucker P, Fischer MR. The questionnaire “SFDP26-German”: a reliable tool for evaluation of clinical teaching? GMS Z Med Ausbild 2011;28(2):Doc30. 35. Wong JG, Agisheva K. Developing teaching skills for medical educators in Russia: a cross-cultural faculty development project. Med Educ. 2007;41(3): 318–24. 36. Zayas TT. Qualities of effective preceptors of physician assistant students. J Phycian Assist Educ. 1999;10(1):7–11. 37. Jarski RW, Kulig K, Olson RE. Allied health perceptions of effective clinical instruction. J Allied Health. 1989;18(5):469–78. 38. Rogers JL, Lautar CJ, Dunn LR. Allied health students’ perceptions of effective clinical instruction. Health Care Manag (Frederick). 2010;29(1):63–7. 39. Bienstock JL, Martin S, Tzou W, Fox HE. Medical students’ gender is a predictor of success in the obstetrics and gynecology basic clerkship. Teach Learn Med. 2002;14(4):240–3. 40. Haist SA, Wilson JF, Elam CL, Blue AV, Fosson SE. The effect of gender and age on medical school performance: an important interaction. Adv Health Sci Educ Theory Pract. 2000;5(3):197–205. 41. Lewisson N, Hellgren L, Johansson J. Quality improvement in clinical teaching through student evaluations of rotations and feedback to departments. Med Teach. 2013;35(10):820–5. 42. Wieland ML, Halvorsen AJ, Chaudhry R, Reed DA, McDonald FS, Thomas KG. An evaluation of internal medicine residency continuity clinic redesign to a 50/50 outpatient-inpatient model. J Gen Intern Med. 2013;28(8):1014–9. 43. Dyrbye LN, Thomas MR, Harper W, Massie FS Jr, Power DV, Eacker A, et al. The learning environment and medical student burnout: a multicentre study. Med Educ. 2009;43(3):274–82. 44. Remmen R, Denekens J, Scherpbier A, Hermann I, van der Vleuten C, Royen PV, et al. An evaluation study of the didactic quality of clerkships. Med Educ. 2000;34(6):460–4. 45. Torre DM, Sebastian JL, Simpson DE. Learning activities and high-quality teaching: perceptions of third-year IM clerkship students. Acad Med. 2003; 78(8):812–4. 46. Schneider JR, Coyle JJ, Ryan ER, Bell RH Jr, DaRosa DA. Implementation and evaluation of a new surgical residency model. J Am Coll Surg. 2007;205(3): 393–404. 47. Poncelet A, O’Brien B. Preparing medical students for clerkships: a descriptive analysis of transition courses. Acad Med. 2008;83(5):444–51. 48. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166.e7–16. 49. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:1002–9. 50. Kirkpatrick D. Revisiting Kirkpatrick's four-level model. Training and. Development. 1996;50:54–9.

Journal

BMC Medical EducationSpringer Journals

Published: Jun 4, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off