Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Correcting the Bias of Clerkship Timing on Academic Performance

Correcting the Bias of Clerkship Timing on Academic Performance ObjectivesTo document the time-of-year bias in National Board of Medical Examiners subject examination (NBME) scores in a third-year pediatrics clerkship and to develop a grading method that neutralizes the bias.DesignInterventional modeling of final grades.SettingUniversity-based medical school.Subjects and MethodsDuring each of the past 3 academic years, we conducted six 2-month pediatric clerkships for third-year students. To counter the time-of-year bias, NBME scores, clinical evaluations, and departmental examination scores for the current rotation were pooled with those from the rotations from the same time of year during the previous 2 years. Final grades for the current rotation were determined by cutoff points derived from that entire 3-year pool. We analyzed this approach by testing the time-of-year effects on NBME scores, clinical evaluations, and final grades while maintaining step 1 of the US Medical Licensing Examination as a preclinical baseline control.ResultsThe scores for step 1 of the US Medical Licensing Examination did not differ significantly by time of year. Clinical evaluations and NBME scores showed significant upward trends as the academic year progressed. By contrast, according to design, final grades showed no significant time-of-year trend.ConclusionsOur results support previous reports of significant improvements in NBME scores across the academic year. Our method of computing final grades corrects for this time-of-year bias by judging students only in relation to those who took the rotation at the same time of year. It is our belief that the prevalence and significance of the time-of-year trend warrants such an adjustment in grading to help minimize the academic disadvantage faced by students early in their clinical training.THE NATIONAL Board of Medical Examiners subject examination (NBME)is used by more than 60% of the pediatrics departments of US medical schools as a component of clerkship grading.The NBME is attractive because it is nationally standardized and is constructed to cover the subject matter broadly.It has become apparent, however, that the NBME presents a challenge for educational measurement: medical educators are reporting significant effects of clerkship timing (ie, time of year) on NBME scores and final grades. Whalen and Mosesfound that student scores on the NBME medicine clerkship examination and final grades significantly increased as the academic year progressed. Baciewicz et aland Ripkey et alreported a similar trend for the NBME surgery clerkship examination. Clark and Jelovsek,Manetta et al,and Smith et alreported that mean NBME obstetrics and gynecology clerkship scores were higher later in the year; and Hampton et alreported the time-of-year trend for NBME obstetrics and gynecology scores and for final grades in obstetrics and gynecology and pediatrics. In contrast to these results, the time-of-year effect on NBME psychiatry scores appears to be very weakor nonexistent.A common speculation about the time-of-year trend is that students coming to a given clerkship later in the year have an advantage because of generalized clinical experience and technical knowledge acquired during that year's previous clerkships.When the NBME score is considered in computing the student's final grade, the time-of-year trend presents an interesting problem to the medical educator and the medical student. From one perspective, the trend may be accepted, and the student may be encouraged to capitalize on it, rather than fall passive victim to "the accident of scheduling."In this vein, Baciewicz et aland Ripkey et alrecommended that students should take their surgery clerkship later in the year if they wish to maximize their performance in that specialty, and Hampton et almaintained that students who aspire to enter obstetrics and gynecology should be aware of the end-of-year grade advantage. Whalen and Mosesconcurred with respect to the internal medicine rotation."Handicapping" of grades and "special compensation" have been mentioned as alternatives to advising students to take advantage of the time-of-year effect.However, no handicapping system seems to have been described. We believe that it can be an actual and unproductive burden on the student to need to play a statistical advantage when specialty choices are far from settled. In our experience, at the end of the second year most students have only a weak idea concerning their ultimate choice of specialty, and strong specialty preferences often develop unexpectedly on particular services long after the accident of rotation scheduling. With this in mind, in the academic year, 1994-1995 we developed a grading system for the pediatrics clerkship that is specifically aimed at neutralizing the time-of-year handicap. We herein report our method and experience during the past 3 years.SUBJECTS AND METHODSSTUDENTSThere were six 2-month pediatrics clerkship rotations (July-August, September-October, November-December, January-February, March-April, and May-June) in each of the following 3 academic years: 1994-1995 (n=129), 1995-1996 (n=120), and 1996-1997 (n=118) (total sample, n=367). The median number of students per rotation was 21 (range, 12-24; interquartile range, 19-22). Students were assigned to rotations according to a long-standing policy that combines student choice and lottery. This may be regarded as a sample of convenience. No information was available concerning factors that may have influenced individual students' assignments to any particular rotation. We have observed informally that very few students had strong specialty preferences going into their third year.MEASURESThe students took step 1 of the US Medical Licensing Examination (USMLE-1)at the end of the second year, before being assigned to any third-year clerkship. The USMLE-1 is highly correlated with NBME scores,and therefore was used as an appropriate preclinical baseline measure to control statistically for possible academic biases associated with time of year.A departmental written examination was administered midway through each rotation. It was worth 15% of the final grade (results not reported herein). At the end of the rotation, the student took the NBME in pediatrics(35% of the final grade). Before announcement of the NBME results, each student underwent subjective evaluation for clinical ability by an average of 3 attending physicians, 3 senior residents, and 2 junior residents. Each evaluator gave a qualitative grade (superior [roughly equivalent to an "A"&rsqb;, high satisfactory ["B"&rsqb;, satisfactory ["C"&rsqb;, low satisfactory ["D"&rsqb;, and unsatisfactory ["F"&rsqb;) in each of the following 4 domains: factual knowledge, practical skill, personal and professional behavior, and values. Numeric scores were assigned for each of the 4 domains and then averaged for each evaluator. The evaluators' scores were averaged within level (attending physician, senior resident, and junior resident), weighted by level, and finally averaged to make up 50% of the final grade.GRADINGTo neutralize the time-of-year trend, we graded the students by pooling their scores with those of all students who had taken the rotation at that same time of year during the past 2 years (or, in cases where data from previous years were not yet sufficiently numerous, the rotation directly before theirs). Each student was thus compared with approximately 60 others, all of whom had taken pediatrics at almost the same time of year. For each measure (departmental examination, NBME score, and clinical evaluation), the approximately 60 scores were standardized to mean (SD) of 86 (5.19). The 3 standardized scores were multiplied by their respective weights (0.15, 0.35, and 0.50), and the 3 weighted scores were summed to determine the final grade. Using the department's traditional numerical grading policy, this standardization would yield approximately 25% grades of superior (A), 50% grades of high satisfactory (B), and 25% grades of satisfactory (C) for the entire group of 60. The actual distribution of grades departed from this ideal for any particular rotation of approximately 20 students, depending on their strengths relative to the entire group with whom they were standardized.STATISTICAL ANALYSISDescriptive statistics and statistical tests were computed using BMDP PC-90.Data are reported as mean (SD). Time-of-year trends are graphed as mean raw scores. For each outcome measure, the effects of year, time of year, and their interaction were tested using 2-way factorial analysis of variance (ANOVA) with Brown-Forsythe adjustments for unequal variances; post hoc follow-up comparisons were performed using Student-Newman-Keuls tests.Specific hypotheses concerning time-of-year trends were tested separately using linear and quadratic contrasts.Correlations among measures were tested using the Pearson product moment correlation r(α=.05 throughout). Complete data were available for all students for all variables except USMLE-1, for which 6 students' data (1.6%) could not be identified because of name change (n=4) or could not be used because of a different scoring system (n=2).RESULTSYear of clerkship was not significantly related to NBME score (P=.26), clinical evaluation (P=.10), final grade (P=.47), or USMLE-1 (P=.08). Therefore, the data were combined for all 3 years and analyzed by time of year.STEP 1 OF USMLEAcross the 6 rotations, mean (SD) USMLE-1 scores were 210 (20), 208 (15), 204 (18), 207 (15), 205 (20), and 207 (17) (Figure 1). There were no significant differences or trends for time of year (for the overall ANOVA and the linear and quadratic trends, all Pvalues >.23). This flat preclinical baseline suggests that time-of-year trends observed in other measures did not result from accidental confounding of time of year with academic ability.Time-of-year trends for National Board of Medical Examiners subject examination (NBME) scores in pediatrics, departmental clinical evaluation, final grade, and step 1 of the US Medical Licensing Examination (USMLE-1) for 367 3rd-year students in a pediatrics clerkship. Significant trends were found only for NBME score (linear trend, P<.001) and clinical evaluation (linear trend, P<.001; quadratic trend, P=.03).NBME SCORESMean (SD) NBME scores for the 6 rotations were 477 (110), 466 (80), 484 (112), 506 (112), 523 (86), and 526 (83) (Figure 1). The increase across time of year was significant (linear trend, P<.001), and post-hoc comparisons showed the first 2 rotations to be significantly lower than the last 2. This ordering of first 2 vs last 2 rotations was found in each of the 3 years separately. It was significant for academic years 1994-1995 (P=.005) and 1995-1996 (P=.002), but not 1996-1997 (P=.11).CLINICAL EVALUATIONSMean (SD) clinical evaluation scores for the 6 rotations were 87.1 (2.9), 87.6 (3.2), 89.3 (2.2), 89.6 (3.6), 89.7 (3.1), and 89.8 (2.9) (Figure 1). The increase across time of year was significant (linear trend, P<.001), but occurred mostly in the early rotations (quadratic trend, P=.03).WEIGHTED FINAL GRADESMean (SD) final grades for the 6 rotations were 86.3 (4.3), 86.4 (4.2), 87.2 (3.6), 86.6 (4.4), 86.5 (4.1), and 86.1 (3.9) (Figure 1). The time-of-year function was flat, showing no significant changes or trends as the academic year progressed (all Pvalues >.21).CORRELATIONS AMONG MEASURESFor the sample as a whole, the 3 outcome measures were significantly intercorrelated. For final grade vs NBME score, r=0.75 (P<.001); for final grade vs clinical evaluation, r=0.78 (P<.001). These correlations will not be discussed further because measures that form part of the final grade will necessarily be correlated with it.The clinical evaluation was independent of NBME, however, so their correlation (r=0.48; P<.001) is of interest.COMMENTData from our pediatrics clerkship confirm the upward time-of-year trend in NBME scores identified in earlier reports for medicine,surgery,and obstetrics and gynecology clerkships.Our findings also support the data of Hampton et al,who reported that NBME pediatrics scores increased toward the end of the year (although nonsignificantly in their data).Few studies have examined the relation of clerkship timing and subjective evaluations. Whalen and Mosesfound no timing effect in the percentage of students receiving outstanding or above-average subjective evaluations in internal medicine.Baciewicz et alreported likewise for oral examinations in surgery. In contrast, our analysis showed a significant timing bias in subjective clinical evaluations that paralleled that of the NBME pediatrics clerkship score. The reasons for the disparity between our findings and those of Whalen and Mosesand Baciewicz et alare unclear, as the contents and criteria of the programs' subjective evaluations cannot be compared with precision. A possible explanation for our findings is that our residents' and attending physicians' evaluations may have emphasized their impressions of the students' factual academic knowledge. This interpretation is supported by the significant correlation found herein between the subjective clinical evaluations and the NBME scores (r=0.48).In contrast to the NBME results, our data on final clerkship grades differed dramatically from previous reports of time-of-year final grade increases in medicine,obstetrics and gynecology,and pediatrics.Our finding of no time-of-year trend in final grades was precisely the desired outcome of our method of calculating grades to neutralize the time-of-year effect. Thus, the approach seems to realize the handicapping concept that was raised hypothetically by Whalen and Moses,Ripkey et al,and Clark and Jelovsekbut apparently not previously implemented.Beginning in the academic year 1994-1995, we have used a grading system that takes into account clerkship timing when calculating final grades. Due to the bias in objective testing and subjective evaluations favoring students who complete their pediatrics clerkships later in the academic year, we scale our final scores so that student performance is considered only in relation to students taking the pediatric clerkship at the same time in their medical training (eg, a student in the January-February rotation of academic year 1996-1997 underwent evaluation relative to the January-February rotations from that year plus the 2 previous years).Given the weight that residency selection committees can put on medical students' final grades, we believe it is the responsibility of clerkship directors and faculty to develop and refine grading systems that are responsive to known biases such as clerkship timing. Simply recommending that a student take his pediatrics or surgery clerkship later in the year if he has a hunch that he would like to become a pediatrician or a surgeon is an unsatisfying response. In our experience, at the end of their second year most students do not have a strong specialty preference; and almost regardless of preference, the rotation order is determined by the whim of computer lottery. Moreover, encouraging a student to play an early hunch fails to recognize the important role that medical educators play in opening doors of interest and opportunity for their students during the required clerkship year. Because our grading method is sensitive to clerkship timing, it effectively minimizes at least one disadvantage faced by students early in their clinical training.National Board of Medical ExaminersSubject Examination Program: General Information and Sample Items.Philadelphia, Pa: National Board of Medical Examiners; 1993.BBarzanskyHSJonasSIEtzelEducational programs in US medical schools, 1995-1996.JAMA.1996;276:714-719.KIHoffmanGuidance for the use of the USMLE in medical education settings, II: the USMLE, the NBME subject examinations, and assessment of individual academic achievement.Acad Med.1993;68:740-747.JPWhalenVKMosesThe effect on grades of the timing and site of third-year internal medicine clerkships.Acad Med.1990;65:708-709.FABaciewiczLArentMWeaverRYeastingsNRThomfordInfluence of clerkship structure and timing on individual student performance.Am J Surg.1990;159:265-268.DRRipkeySMCaseDBSwansonPredicting performance on the NBME surgery subject test and USMLE Step 2: the effects of surgery clerkship timing and length.Acad Med.1997;72(suppl 1):531-533.KHClarkFRJelovsekEffect of clerkship timing on third-year medical students' grades and NBME scores in an obstetrics-gynecology clerkship.Acad Med.1992;67:865.AManettaEManettaDEmmaKAKeeganJHWilliamsEffects of rotation discipline on medical student grades in obstetrics and gynecology throughout the academic year.Am J Obstet Gynecol.1993;169:1215-1217.ERSmithTVDinhGAndersonA decrease from 8 to 6 weeks in obstetrics and gynecology clerkship: effect on medical students' cognitive knowledge.Obstet Gynecol.1995;86:458-460.HLHamptonBJCollinsKGPerryEFMeydrechWLWiserJCMorrisonOrder of rotation in third-year clerkships: influence on academic performance.J Reprod Med.1996;41:337-340.SMCaseDRRipkeyDBSwansonThe effects of psychiatry clerkship timing and length on measures of performance.Acad Med.1997;72(suppl 1):534-536.JPWhalenInvestigating whether timing of students' third-year internal medicine clerkships affects their performances as seniors on the NBME examination.Acad Med.1991;11:709.Federation of State Medical Boards of the US, Inc, and the National Board of Medical ExaminersUSMLE Step 1 General Instructions, Content Description, and Sample Items.Philadelphia, Pa: National Board of Medical Examiners; 1996.GFerenchickBMavisJO'DonnellSSmithMLoehrkePLambertThe effect of testing time on students' performance for the subject examination for internal medicine.J Gen Intern Med.1995;10:151-153.WJDixonMBBrownLEngelmanRIJennrichBMDP Statistical Software Manual.Los Angeles: University of California; 1990;1:180-212, 548.BJWinerStatistical Principles in Experimental Design.2nd ed. New York, NY: McGraw-Hill Book Co; 1971:177-185.Accepted for publication April 30, 1998.We thank Dwayne A. Ollerich, PhD, for assistance in USMLE-1 data acquisition.Reprints: Cheng T. Cho, MD, PhD, Department of Pediatrics, University of Kansas Medical Center, 3901 Rainbow Blvd, Kansas City, KS 66160.Editor's Note:I'm all for anything that leads to fairer evaluations of anyone. However, I still remain concerned about the emphasis placed on grades in what's supposed to be adult education.—Catherine D. DeAngelis, MD http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JAMA Pediatrics American Medical Association

Correcting the Bias of Clerkship Timing on Academic Performance

Cho, Julie E.; Belmont, John M.; Cho, Cheng T.
JAMA Pediatrics , Volume 152 (10) – Oct 1, 1998
Download PDF
4 pages

Loading next page...
 
/lp/american-medical-association/correcting-the-bias-of-clerkship-timing-on-academic-performance-QcsWZ7O48N

References (20)

Publisher
American Medical Association
Copyright
Copyright 1998 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.
ISSN
2168-6203
eISSN
2168-6211
DOI
10.1001/archpedi.152.10.1015
Publisher site
See Article on Publisher Site

Abstract

ObjectivesTo document the time-of-year bias in National Board of Medical Examiners subject examination (NBME) scores in a third-year pediatrics clerkship and to develop a grading method that neutralizes the bias.DesignInterventional modeling of final grades.SettingUniversity-based medical school.Subjects and MethodsDuring each of the past 3 academic years, we conducted six 2-month pediatric clerkships for third-year students. To counter the time-of-year bias, NBME scores, clinical evaluations, and departmental examination scores for the current rotation were pooled with those from the rotations from the same time of year during the previous 2 years. Final grades for the current rotation were determined by cutoff points derived from that entire 3-year pool. We analyzed this approach by testing the time-of-year effects on NBME scores, clinical evaluations, and final grades while maintaining step 1 of the US Medical Licensing Examination as a preclinical baseline control.ResultsThe scores for step 1 of the US Medical Licensing Examination did not differ significantly by time of year. Clinical evaluations and NBME scores showed significant upward trends as the academic year progressed. By contrast, according to design, final grades showed no significant time-of-year trend.ConclusionsOur results support previous reports of significant improvements in NBME scores across the academic year. Our method of computing final grades corrects for this time-of-year bias by judging students only in relation to those who took the rotation at the same time of year. It is our belief that the prevalence and significance of the time-of-year trend warrants such an adjustment in grading to help minimize the academic disadvantage faced by students early in their clinical training.THE NATIONAL Board of Medical Examiners subject examination (NBME)is used by more than 60% of the pediatrics departments of US medical schools as a component of clerkship grading.The NBME is attractive because it is nationally standardized and is constructed to cover the subject matter broadly.It has become apparent, however, that the NBME presents a challenge for educational measurement: medical educators are reporting significant effects of clerkship timing (ie, time of year) on NBME scores and final grades. Whalen and Mosesfound that student scores on the NBME medicine clerkship examination and final grades significantly increased as the academic year progressed. Baciewicz et aland Ripkey et alreported a similar trend for the NBME surgery clerkship examination. Clark and Jelovsek,Manetta et al,and Smith et alreported that mean NBME obstetrics and gynecology clerkship scores were higher later in the year; and Hampton et alreported the time-of-year trend for NBME obstetrics and gynecology scores and for final grades in obstetrics and gynecology and pediatrics. In contrast to these results, the time-of-year effect on NBME psychiatry scores appears to be very weakor nonexistent.A common speculation about the time-of-year trend is that students coming to a given clerkship later in the year have an advantage because of generalized clinical experience and technical knowledge acquired during that year's previous clerkships.When the NBME score is considered in computing the student's final grade, the time-of-year trend presents an interesting problem to the medical educator and the medical student. From one perspective, the trend may be accepted, and the student may be encouraged to capitalize on it, rather than fall passive victim to "the accident of scheduling."In this vein, Baciewicz et aland Ripkey et alrecommended that students should take their surgery clerkship later in the year if they wish to maximize their performance in that specialty, and Hampton et almaintained that students who aspire to enter obstetrics and gynecology should be aware of the end-of-year grade advantage. Whalen and Mosesconcurred with respect to the internal medicine rotation."Handicapping" of grades and "special compensation" have been mentioned as alternatives to advising students to take advantage of the time-of-year effect.However, no handicapping system seems to have been described. We believe that it can be an actual and unproductive burden on the student to need to play a statistical advantage when specialty choices are far from settled. In our experience, at the end of the second year most students have only a weak idea concerning their ultimate choice of specialty, and strong specialty preferences often develop unexpectedly on particular services long after the accident of rotation scheduling. With this in mind, in the academic year, 1994-1995 we developed a grading system for the pediatrics clerkship that is specifically aimed at neutralizing the time-of-year handicap. We herein report our method and experience during the past 3 years.SUBJECTS AND METHODSSTUDENTSThere were six 2-month pediatrics clerkship rotations (July-August, September-October, November-December, January-February, March-April, and May-June) in each of the following 3 academic years: 1994-1995 (n=129), 1995-1996 (n=120), and 1996-1997 (n=118) (total sample, n=367). The median number of students per rotation was 21 (range, 12-24; interquartile range, 19-22). Students were assigned to rotations according to a long-standing policy that combines student choice and lottery. This may be regarded as a sample of convenience. No information was available concerning factors that may have influenced individual students' assignments to any particular rotation. We have observed informally that very few students had strong specialty preferences going into their third year.MEASURESThe students took step 1 of the US Medical Licensing Examination (USMLE-1)at the end of the second year, before being assigned to any third-year clerkship. The USMLE-1 is highly correlated with NBME scores,and therefore was used as an appropriate preclinical baseline measure to control statistically for possible academic biases associated with time of year.A departmental written examination was administered midway through each rotation. It was worth 15% of the final grade (results not reported herein). At the end of the rotation, the student took the NBME in pediatrics(35% of the final grade). Before announcement of the NBME results, each student underwent subjective evaluation for clinical ability by an average of 3 attending physicians, 3 senior residents, and 2 junior residents. Each evaluator gave a qualitative grade (superior [roughly equivalent to an "A"&rsqb;, high satisfactory ["B"&rsqb;, satisfactory ["C"&rsqb;, low satisfactory ["D"&rsqb;, and unsatisfactory ["F"&rsqb;) in each of the following 4 domains: factual knowledge, practical skill, personal and professional behavior, and values. Numeric scores were assigned for each of the 4 domains and then averaged for each evaluator. The evaluators' scores were averaged within level (attending physician, senior resident, and junior resident), weighted by level, and finally averaged to make up 50% of the final grade.GRADINGTo neutralize the time-of-year trend, we graded the students by pooling their scores with those of all students who had taken the rotation at that same time of year during the past 2 years (or, in cases where data from previous years were not yet sufficiently numerous, the rotation directly before theirs). Each student was thus compared with approximately 60 others, all of whom had taken pediatrics at almost the same time of year. For each measure (departmental examination, NBME score, and clinical evaluation), the approximately 60 scores were standardized to mean (SD) of 86 (5.19). The 3 standardized scores were multiplied by their respective weights (0.15, 0.35, and 0.50), and the 3 weighted scores were summed to determine the final grade. Using the department's traditional numerical grading policy, this standardization would yield approximately 25% grades of superior (A), 50% grades of high satisfactory (B), and 25% grades of satisfactory (C) for the entire group of 60. The actual distribution of grades departed from this ideal for any particular rotation of approximately 20 students, depending on their strengths relative to the entire group with whom they were standardized.STATISTICAL ANALYSISDescriptive statistics and statistical tests were computed using BMDP PC-90.Data are reported as mean (SD). Time-of-year trends are graphed as mean raw scores. For each outcome measure, the effects of year, time of year, and their interaction were tested using 2-way factorial analysis of variance (ANOVA) with Brown-Forsythe adjustments for unequal variances; post hoc follow-up comparisons were performed using Student-Newman-Keuls tests.Specific hypotheses concerning time-of-year trends were tested separately using linear and quadratic contrasts.Correlations among measures were tested using the Pearson product moment correlation r(α=.05 throughout). Complete data were available for all students for all variables except USMLE-1, for which 6 students' data (1.6%) could not be identified because of name change (n=4) or could not be used because of a different scoring system (n=2).RESULTSYear of clerkship was not significantly related to NBME score (P=.26), clinical evaluation (P=.10), final grade (P=.47), or USMLE-1 (P=.08). Therefore, the data were combined for all 3 years and analyzed by time of year.STEP 1 OF USMLEAcross the 6 rotations, mean (SD) USMLE-1 scores were 210 (20), 208 (15), 204 (18), 207 (15), 205 (20), and 207 (17) (Figure 1). There were no significant differences or trends for time of year (for the overall ANOVA and the linear and quadratic trends, all Pvalues >.23). This flat preclinical baseline suggests that time-of-year trends observed in other measures did not result from accidental confounding of time of year with academic ability.Time-of-year trends for National Board of Medical Examiners subject examination (NBME) scores in pediatrics, departmental clinical evaluation, final grade, and step 1 of the US Medical Licensing Examination (USMLE-1) for 367 3rd-year students in a pediatrics clerkship. Significant trends were found only for NBME score (linear trend, P<.001) and clinical evaluation (linear trend, P<.001; quadratic trend, P=.03).NBME SCORESMean (SD) NBME scores for the 6 rotations were 477 (110), 466 (80), 484 (112), 506 (112), 523 (86), and 526 (83) (Figure 1). The increase across time of year was significant (linear trend, P<.001), and post-hoc comparisons showed the first 2 rotations to be significantly lower than the last 2. This ordering of first 2 vs last 2 rotations was found in each of the 3 years separately. It was significant for academic years 1994-1995 (P=.005) and 1995-1996 (P=.002), but not 1996-1997 (P=.11).CLINICAL EVALUATIONSMean (SD) clinical evaluation scores for the 6 rotations were 87.1 (2.9), 87.6 (3.2), 89.3 (2.2), 89.6 (3.6), 89.7 (3.1), and 89.8 (2.9) (Figure 1). The increase across time of year was significant (linear trend, P<.001), but occurred mostly in the early rotations (quadratic trend, P=.03).WEIGHTED FINAL GRADESMean (SD) final grades for the 6 rotations were 86.3 (4.3), 86.4 (4.2), 87.2 (3.6), 86.6 (4.4), 86.5 (4.1), and 86.1 (3.9) (Figure 1). The time-of-year function was flat, showing no significant changes or trends as the academic year progressed (all Pvalues >.21).CORRELATIONS AMONG MEASURESFor the sample as a whole, the 3 outcome measures were significantly intercorrelated. For final grade vs NBME score, r=0.75 (P<.001); for final grade vs clinical evaluation, r=0.78 (P<.001). These correlations will not be discussed further because measures that form part of the final grade will necessarily be correlated with it.The clinical evaluation was independent of NBME, however, so their correlation (r=0.48; P<.001) is of interest.COMMENTData from our pediatrics clerkship confirm the upward time-of-year trend in NBME scores identified in earlier reports for medicine,surgery,and obstetrics and gynecology clerkships.Our findings also support the data of Hampton et al,who reported that NBME pediatrics scores increased toward the end of the year (although nonsignificantly in their data).Few studies have examined the relation of clerkship timing and subjective evaluations. Whalen and Mosesfound no timing effect in the percentage of students receiving outstanding or above-average subjective evaluations in internal medicine.Baciewicz et alreported likewise for oral examinations in surgery. In contrast, our analysis showed a significant timing bias in subjective clinical evaluations that paralleled that of the NBME pediatrics clerkship score. The reasons for the disparity between our findings and those of Whalen and Mosesand Baciewicz et alare unclear, as the contents and criteria of the programs' subjective evaluations cannot be compared with precision. A possible explanation for our findings is that our residents' and attending physicians' evaluations may have emphasized their impressions of the students' factual academic knowledge. This interpretation is supported by the significant correlation found herein between the subjective clinical evaluations and the NBME scores (r=0.48).In contrast to the NBME results, our data on final clerkship grades differed dramatically from previous reports of time-of-year final grade increases in medicine,obstetrics and gynecology,and pediatrics.Our finding of no time-of-year trend in final grades was precisely the desired outcome of our method of calculating grades to neutralize the time-of-year effect. Thus, the approach seems to realize the handicapping concept that was raised hypothetically by Whalen and Moses,Ripkey et al,and Clark and Jelovsekbut apparently not previously implemented.Beginning in the academic year 1994-1995, we have used a grading system that takes into account clerkship timing when calculating final grades. Due to the bias in objective testing and subjective evaluations favoring students who complete their pediatrics clerkships later in the academic year, we scale our final scores so that student performance is considered only in relation to students taking the pediatric clerkship at the same time in their medical training (eg, a student in the January-February rotation of academic year 1996-1997 underwent evaluation relative to the January-February rotations from that year plus the 2 previous years).Given the weight that residency selection committees can put on medical students' final grades, we believe it is the responsibility of clerkship directors and faculty to develop and refine grading systems that are responsive to known biases such as clerkship timing. Simply recommending that a student take his pediatrics or surgery clerkship later in the year if he has a hunch that he would like to become a pediatrician or a surgeon is an unsatisfying response. In our experience, at the end of their second year most students do not have a strong specialty preference; and almost regardless of preference, the rotation order is determined by the whim of computer lottery. Moreover, encouraging a student to play an early hunch fails to recognize the important role that medical educators play in opening doors of interest and opportunity for their students during the required clerkship year. Because our grading method is sensitive to clerkship timing, it effectively minimizes at least one disadvantage faced by students early in their clinical training.National Board of Medical ExaminersSubject Examination Program: General Information and Sample Items.Philadelphia, Pa: National Board of Medical Examiners; 1993.BBarzanskyHSJonasSIEtzelEducational programs in US medical schools, 1995-1996.JAMA.1996;276:714-719.KIHoffmanGuidance for the use of the USMLE in medical education settings, II: the USMLE, the NBME subject examinations, and assessment of individual academic achievement.Acad Med.1993;68:740-747.JPWhalenVKMosesThe effect on grades of the timing and site of third-year internal medicine clerkships.Acad Med.1990;65:708-709.FABaciewiczLArentMWeaverRYeastingsNRThomfordInfluence of clerkship structure and timing on individual student performance.Am J Surg.1990;159:265-268.DRRipkeySMCaseDBSwansonPredicting performance on the NBME surgery subject test and USMLE Step 2: the effects of surgery clerkship timing and length.Acad Med.1997;72(suppl 1):531-533.KHClarkFRJelovsekEffect of clerkship timing on third-year medical students' grades and NBME scores in an obstetrics-gynecology clerkship.Acad Med.1992;67:865.AManettaEManettaDEmmaKAKeeganJHWilliamsEffects of rotation discipline on medical student grades in obstetrics and gynecology throughout the academic year.Am J Obstet Gynecol.1993;169:1215-1217.ERSmithTVDinhGAndersonA decrease from 8 to 6 weeks in obstetrics and gynecology clerkship: effect on medical students' cognitive knowledge.Obstet Gynecol.1995;86:458-460.HLHamptonBJCollinsKGPerryEFMeydrechWLWiserJCMorrisonOrder of rotation in third-year clerkships: influence on academic performance.J Reprod Med.1996;41:337-340.SMCaseDRRipkeyDBSwansonThe effects of psychiatry clerkship timing and length on measures of performance.Acad Med.1997;72(suppl 1):534-536.JPWhalenInvestigating whether timing of students' third-year internal medicine clerkships affects their performances as seniors on the NBME examination.Acad Med.1991;11:709.Federation of State Medical Boards of the US, Inc, and the National Board of Medical ExaminersUSMLE Step 1 General Instructions, Content Description, and Sample Items.Philadelphia, Pa: National Board of Medical Examiners; 1996.GFerenchickBMavisJO'DonnellSSmithMLoehrkePLambertThe effect of testing time on students' performance for the subject examination for internal medicine.J Gen Intern Med.1995;10:151-153.WJDixonMBBrownLEngelmanRIJennrichBMDP Statistical Software Manual.Los Angeles: University of California; 1990;1:180-212, 548.BJWinerStatistical Principles in Experimental Design.2nd ed. New York, NY: McGraw-Hill Book Co; 1971:177-185.Accepted for publication April 30, 1998.We thank Dwayne A. Ollerich, PhD, for assistance in USMLE-1 data acquisition.Reprints: Cheng T. Cho, MD, PhD, Department of Pediatrics, University of Kansas Medical Center, 3901 Rainbow Blvd, Kansas City, KS 66160.Editor's Note:I'm all for anything that leads to fairer evaluations of anyone. However, I still remain concerned about the emphasis placed on grades in what's supposed to be adult education.—Catherine D. DeAngelis, MD

Journal

JAMA Pediatrics – American Medical Association

Published: Oct 1, 1998

There are no references for this article.