A Longitudinal Study of Commonly Used Admissions Measures and Disenrollment from Medical School and Graduate Medical Education Probation or Termination from Training

A Longitudinal Study of Commonly Used Admissions Measures and Disenrollment from Medical School... Abstract Introduction This is an empirical study to better understand commonly used medical school admission measures and disenrollment decisions during undergraduate medical education as well as graduate medical education (GME) probation or termination decisions. Materials and Methods Based on the data of USUHS medical students matriculating between 1998 and 2011 (N = 2,460), we compared medical school graduates and those disenrolled from medical school on MCAT scores, undergraduate BCPM (Biology, Chemistry, Physics, Math) GPA, and undergraduate overall GPA. We also reported more specific reasons for disenrollment decisions. Next, we compared the students who were referred to the student promotion committee (SPC) with other students on these measures. Moving onto GME, we compared trainees who were put on probation or terminated from training with those who were not on MCAT and undergraduate GPA measures. In addition, we examined the association between being referred to the SPC and GME probation or termination. Results There were 2,347 graduates and 113 disenrolled students from medical school (4.8%). For the disenrolled students, 43 (38.7%) students were disenrolled for exclusively (or primarily) non-academic reasons, and 68 (61.3%) were disenrolled for exclusively (or primarily) academic reasons. The t-tests showed statistically significant differences on the MCAT score of the first attempt (t(2,449) = 7.22, P < 0.01, Cohen’s d = 0.70), average MCAT score (t(2,449) = 4.22, P < 0.01, Cohen’s d = 0.41), and highest MCAT score (t(2,449) = 3.51, P < 0.01, Cohen’s d = 0.34). Logistic regression model selection also revealed that the best predictor for disenrollment was the first MCAT score (exp(b) = 0.83, 95% CI = (0.78, 0.88)). No significant differences on these measures were found from the group comparisons on SPC and GME probation or termination. There was no significant association between SPC appearance and GME probation or termination. Conclusions Academic difficulties, especially in the basic sciences, appear to be the most common factor for disenrollment from medical school. These students also had lower MCAT scores, particularly on the first attempt. The MCAT performance indicators and undergraduate GPA were consistently lower, but not statistically significant, for those who appeared before SPC or were put on probation or terminated from training during GME. INTRODUCTION Over the past 30 years, the graduation rate for medical students has decreased. According to a recent AAMC’s report,1 the 4-yr graduation rate of U.S. medical school students was 82.5% and the 5-yr graduation rate was 94.1%. Further, many scholars have argued that the United States is currently facing a physician shortage based on projected health care needs for society2,3 and undergraduate medical education (UME) disenrollment as well as graduate medical education (GME) probation or termination from training can contribute to the reduction in physician supply. Additionally, learners who are disenrolled from medical school or placed on probation or terminated from training during GME result in significant financial, time and other resource costs to both the institution and the learner. Thus, work is needed to better understand commonly used admissions measures and their association with disenrollment decisions at UME as well as GME probation or termination. Studies on disenrollment must rely on access to large amounts of longitudinal data, often from a single institution, given that UME disenrollment decisions and GME probation or termination are quite rare in the medical profession. Prior studies investigating the prediction of the attrition rates and/or other adverse actions in medical education have focused on either medical school or graduate medical education, but not both.4,5 The present study reports an investigation at the Uniformed Services University of the Health Sciences (USUHS). Upon matriculation, students are commissioned as officers in the Uniformed Services (Army, Navy, Air Force, and Public Health Service) and after graduation, the vast majority train in military-affiliated programs across the country. In this essentially “closed” system, the school can track the students’ performance from admission through all 4 years of medical school and into graduate medical education programs and beyond. A prior study of the attrition of matriculating USUHS medical students enrolled during the years 1980–1999 demonstrated that MCAT scores of disenrolled students (mean = 28.5) were significantly lower than that of those who graduated or were currently enrolled (mean = 31). In addition, the undergraduate GPA was slightly lower in the disenrolled group (mean = 3.41) as compared to the graduated or currently enrolled group (mean = 3.46), but the difference was not statistically significant.6 The purpose of the present study was to seek evidence to answer the research questions that centered around the comparisons of MCAT scores (score of the first attempt, score of the last attempt, average score of all attempts, the highest score of all attempts, and number of attempts) and undergraduate GPAs between medical school graduates and the disenrolled students as well as between GME trainees who were placed on probation or termination and those who were not. We used the MCAT and undergraduate GPAs in part because these are commonly used measures by all medical schools in the USA and thus may provide useful information for purposes of graduation rate questions. We investigated the following research questions. First, do disenrolled medical students differ from graduates on MCAT and undergraduate GPA? Second, what are the reasons for disenrollment from medical school? Third, do students who appeared before the SPC differ from the non-SPC students on MCAT performance or undergraduate GPA? Fourth, do the students who were put on probation or termination during internship, residency, and/or fellowship differ from those who did not on MCAT performance and undergraduate GPA? Lastly, is there an association between SPC appearance and being put on probation or termination during GME? METHODS Participants and measures USUHS is the only federal medical university in the USA. It matriculates approximately 170 students per year into the School of Medicine. The study cohort was medical students matriculating between 1998 and 2011 (N = 2,460). This is the cohort that we had full access to their MCAT scores, undergraduate GPAs, reasons of disenrollment, the Student Promotion Committee (SPC) indicator, and the GME probation or termination indicator. In this study, GME trainees include interns, residents, or fellows. For the MCAT scores, we collected or calculated five measures – score of the first attempt, score of the last attempt, average score of all attempts, the highest score of all attempts, and number of attempts. For the undergraduate GPAs, we collected data on both undergraduate BCPM (Biology, Chemistry, Physics, Math) GPA and the overall GPA. We collected the data on disenrollment as well as reasons for disenrollment which are maintained and updated by the Office of Student Affairs and the SPC. The research team reviewed the reasons for disenrollment and determined the categories and sub-categories of these reasons. The SPC is the official School of Medicine body that reviews matters of serious concern for violations of personal integrity and professional comportment in addition to poor academic performance. The School of Medicine charges the SPC with the responsibility for reviewing cognitive and non-cognitive performance. The committee consists of 10 faculty members appointed by the Dean of the School of Medicine, 2 faculty nominated by the faculty senate, and 2 students. Detailed information about the SPC, such as who were on the committee and what was the criterion of students being referred to SPC, can be found on USUHS’ website.7 We collected data from USUHS graduates who were trained at one of the GME programs administered by the National Capital Consortium (NCC) regarding probation or termination training outcomes. The NCC includes Walter Reed National Military Medical Center, Malcolm Grow Clinics and Surgical Center, and Fort Belvoir Community Hospital. Data Analyses We conducted t-tests to examine whether the mean differences of the various MCAT measures and undergraduate GPA measures were statistically significant between groups as described in the first, third, and fourth research questions. We applied two-tailed t-tests throughout. As the sample size in the graduates’ group was considerably larger than the disenrolled students group, we took a random smaller sample of the graduates group and reran the t-tests. In the results section, we only report the numbers calculated from the whole sample because the results of using a random smaller sample were aligned with the entire sample. Similarly, t-tests were conducted to investigate the differences between students who appeared before the SPC and other students, and those GME trainees who were put on probation or termination with other GME trainees. In addition, we performed contingency table analysis to examine whether appearing before the SPC was related to GME probation or termination to answer the last research question. In the Results section, we report the values of the inferential test statistics if they were statistically significant. Otherwise, we report only descriptive statistics. Results Medical School Graduates vs. Disenrolled Students: MCAT and Undergraduate GPA Comparison In the study cohort, there were 2,347 medical school graduates and 113 (4.8%) disenrolled students from medical school. The comparison of the means of the MCAT scores between the graduates and the disenrolled students is shown in Figure 1. The t-tests using the entire sample showed statistical differences on the MCAT score of the first attempt (graduates = 29.43, disenrolled students = 27.04, t(2,449) = 7.22, P < 0.01, Cohen’s d = 0.70), the average MCAT score (graduates = 29.14, disenrolled students = 27.79, t(2449) = 4.22, P < 0.01, Cohen’s d = 0.41), and the highest MCAT score (graduates = 29.78, disenrolled students = 28.73, t(2,449) = 3.51, P < 0.01, Cohen’s d = 0.34). When we compared the number of MCAT attempts, 1,449 (63.9%) of the graduates and 56 (53.3%) of the disenrolled students made one attempt only; 770 (32.8%) graduates and 42 (40.0%) of the disenrolled students made two attempts. The t-tests on the undergraduate BCPM GPA (mean of graduates = 3.48, mean of disenrolled students = 3.44, t(2449) = 1.45, P > 0.05) and the overall GPA (mean of graduates = 3.52, mean of disenrolled students = 3.51, t(2,449) = 0.40, P > .05) did not reveal significant differences between the groups. FIGURE 1. View largeDownload slide The sample mean of the MCAT score measures of the graduates group and the entire disenrolled students group. Cohen’s d was added to comparisons that yielded significant results from the t-tests. FIGURE 1. View largeDownload slide The sample mean of the MCAT score measures of the graduates group and the entire disenrolled students group. Cohen’s d was added to comparisons that yielded significant results from the t-tests. Reasons of Disenrollment For the disenrolled students, 43 (38.7%) students were disenrolled for exclusively (or primarily) non-academic reasons, and 68 (61.3%) were disenrolled for exclusively (or primarily) academic reasons. We were able to identify the specific reasons for disenrollment for 76 students, among which were academic issues (55) and non-academic issues (21). The more detailed breakdown of disenrollment reasons is displayed in Table I. If we focus on those students disenrolled for academic reasons, the means were 26.95 for the first attempt, 28.15 for the average score, 29.39 for the last attempt, and 29.39 for the highest MCAT attempt. The means of the BCPM undergraduate GPA and the overall undergraduate GPA were 3.41 and 3.48. The comparison with the graduates yielded significant results on the first MCAT score (t(2,406) = 4.74, P < 0.01, Cohen’s d = 0.68) and the average MCAT score (t(2,406) = 4.74, P < 0.05, Cohen’s d = 0.31). Table I. Summary of Reasons of Disenrollment from Medical School (N = 76) Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Table I. Summary of Reasons of Disenrollment from Medical School (N = 76) Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Students Referred to SPC vs. Others: MCAT and Undergraduate GPA Comparison Among the medical school graduates, 319 students (13.6%) appeared before the SPC and 2,028 (86.4%) did not. The sample means of the MCAT score measures and undergraduate GPA measures of the SPC group were consistently lower than the no SPC group but the differences were not statistically significant (see Table II). Table II. The Means and Standard Deviations (in Parenthesis) of the MCAT Measures and Undergraduate GPA Measures of the Students Who Were Recommended to SPC vs. Those Were not Recommended and Trainees Who Were Placed on Probation or Termination during GME vs. Those Who Were Not   MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)    MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)  Table II. The Means and Standard Deviations (in Parenthesis) of the MCAT Measures and Undergraduate GPA Measures of the Students Who Were Recommended to SPC vs. Those Were not Recommended and Trainees Who Were Placed on Probation or Termination during GME vs. Those Who Were Not   MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)    MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)  Trainees Put on Probation or Termination vs. Others: MCAT and Undergraduate GPA Comparison Within the study cohort, 495 graduates entered graduate medical education at the NCC. Among these, we identified 26 trainees (5.3%) who were put on probation or termination during GME. The sample means and standard deviations of the MCAT score measures and BCPM comparison are also displayed in Table II. The differences between the groups (SPC vs. non-SPC and GME probation or termination vs. no GME probation or termination) were not statistically significant. However, the sample means of the MCAT score measures and undergraduate GPA measures were consistently lower than for those trainees who were put on probation or termination during GME. Logistic Regression Model Selection and Fit We conducted logistic regression model selection for the outcomes of disenrollment and GME probation or termination. From the explanatory variables of MCAT scores and undergraduate GPAs, the model selection results indicated that the best predictor for graduates vs. disenrolled students was the first MCAT score (exp(b) = 0.83, 95% CI = (0.78, 0.88)). With every one point increase in the first MCAT score, the odds of getting disenrolled decreased by 17%. We did not observe any significant predictor for the measure of trainees being put on probation or termination from training. Association Between SPC Measure and GME Probation/Termination The cross tabulation between the SPC and GME probation or termination indicated 375 with no SPC and no GME probation or termination, 20 with no SPC but GME probation or termination, 52 with SPC but no GME probation or termination, and 6 with both SPC and GME probation or termination. The risk of GME probation or termination was about two times greater for the SPC group than the no SPC group (the absolute risk of GME probation or termination of the SPC group = 0.10, the absolute risk of GME probation or termination of the non-SPC group = 0.05). The chi-square test of independence did not show significant association between these two measures (X2(1) = 0.2.61; P = 0.11). DISCUSSION Based on 14 years of data, we conducted a study to find evidence-based answers to several research questions that are relevant to both UME and GME. In terms of our first research question, disenrolled medical students showed consistently lower MCAT scores and undergraduate GPAs (both BCPM GPA and the overall GPA) than the graduates although the comparison on GPAs did not reach statistical significance. The difference was most obvious in the first attempt and the average MCAT scores. Our data suggest that these two MCAT measures may be the most important MCAT variables to consider if the goal is to minimize disenrollment rates. This finding was aligned with a previous study by Zhao et al. in which the authors concluded that the average MCAT score is the best measure when predicting USMLE Step 1 score from repeated MCAT scores.8 Given the range of MCAT scores for both disenrolled and non-disenrolled students, an MCAT threshold score to avoid disenrollment could not be determined. Returning to our second research question, the majority (72.3%) of the students were disenrolled for academic reasons. This is consistent with a systematic review of medical school drop outs that found that academic issues were most strongly associated with dropping out of medical school and, by contrast, there was no consistent pattern of demographic variables that predicted drop-out.9 Only about 5% of the students in our study were disenrolled because of non-academic reasons, including one or more professionalism issues, a medical problem, or resignation. It seems that academic challenges, either from basic sciences or clinical studies, should be medical educators’ primary focus when it comes to reducing the disenrollment rate of medical school. Comparing basic sciences and clinical sciences, we found that the former accounted for considerably higher (53.9% in this study) percentage of disenrollment than the latter (18.4%). Our study also suggested that efforts on helping academically challenged students get through the basic sciences education period seem to be the most important factor to consider in terms of reducing disenrollment. Research on identifying and providing remediation for students who struggled in the pre-clerkship years have been reported.10 Further, it should be noted that in this study approximately 20% of the 76 drop outs of medical school resigned for unknown reasons. This group should be further investigated in future studies. We also asked if students who appeared before the SPC differ from the non- SPC students on MCAT performance or undergraduate GPA. MCAT performance indicators and undergraduate GPA were lower for the students who appeared before SPC and the trainees who were put on probation or termination during GME than those who did not appear before the SPC or were put on probation or termination during GME, respectively, but none of these comparisons were statistically significant. This is consistent with the notion that clinical performance is only partly dependent upon medical knowledge. Similarly, preventing GME probation or termination from training was less clear. Comparing GME trainees who were put on probation or termination with those who were not did not yield statistically significant results. Notably, we did not observe significant associations between appearing before the SPC in medical school and GME probation or termination. These data suggest that commonly used measurements in medical school may not predict GME probation or termination. Our past research found that other indicators such as unsuccessful clinical performance which required remediation during the internal medicine clerkship is strongly associated with poor performance at the internship level.11 Our study had several limitations. First, our findings were based on the sample from a single federal medical school. The findings may not be generalizable to other civilian institutions. Second, we focused on the MCAT scores and undergraduate GPA in this study and the MCAT data used were the “old” MCAT and not reflective of the “new” MCAT scoring system that was changed in 2015. Further, we were not able to collect the GME probation or termination data of all the USU graduates – our findings were based on our medical school graduates who entered GME at the NCC, which may not be representative of our graduates as a whole. Additionally, in the present study we did not explore the specific reasons of SPC referral or GME probation or termination. Lastly, we can only infer association rather than causation and we did not explore the impact of remediation on disenrollment. Strengths of our study included the large cohort over many years and a relatively complete database. Additionally, our work spanned from medical school into graduate medical education. CONCLUSION This large-sample longitudinal study across undergraduate medical education and graduate medical education revealed that academic difficulties appear to be the most common factor for students dropping out of medical school. These students typically have lower MCAT scores, particularly the first MCAT attempt score. The students who appeared before the SPC during UME or had GME probation or termination had consistently lower MCAT score and undergraduate GPA than those who did not, but these comparisons were not statistically significant. Thus, basic science knowledge, especially as measured by performance on the Medical College Admissions Test (MCAT), should be recognized as an important component in the process of reducing the attrition of medical students. References 1 Association of American Medical Colleges: Graduation rates and attrition factors in U.S. medical school students. Anal Brief  2014; 14( 5): 1– 2. https://www.aamc.org/download/379220/data/may2014aib-graduationratesandattritionfactorsforusmedschools.pdf. 2 Kirch DG, Petelle K: Addressing the physician shortage: the peril of ignoring demography. JAMA  2017; 317( 19): 1947– 8. doi:10.1001/jama.2017.2714. Google Scholar CrossRef Search ADS PubMed  3 The complexities of physician supply and demand: Projections from 2013 to 2025 (Final Report). Prepared for Association of American Medical Colleges. Available at https://www.aamc.org/download/426242/data/ihsreportdownload.pdf; accessed August 1, 2017. 4 Caulfield M, Redden G, Sondheimer H: Graduation rates and attrition factors for U.S. medical school students. Anal Brief  2014; 14( 5). https://www.aamc.org/download/379220/data/may2014aib-graduationratesandattritionfactorsforusmedschools.pdf. 5 Durning SJ, Dong T, Hemmer PA, et al.  : Are commonly used premedical school or medical school measures associated with Board certification? Mil Med  2015; 180( 4 Suppl): 18– 23. Google Scholar CrossRef Search ADS PubMed  6 Stetto JE, Gackstetter GD, Cruess DF, Hooper TI: Variables associated with attrition from Uniformed Services University of the Health Sciences Medical School. Mil Med  2004; 2: 102– 7. Google Scholar CrossRef Search ADS   7 Uniformed Services University: 2018. Usuhsedu. Available at: https://www.usuhs.edu/students/campuslife/stuaffairsresources; accessed February 16, 2018. 8 Zhao X, Oppler S, Dunleavy D, Kroopnick M: Validity of four approaches of using repeaters' MCAT scores in medical school admissions to predict USMLE step 1 total scores. Acad Med  2010; 85: S64– 7. doi:10.1097/acm.0b013e3181ed38fc. Google Scholar CrossRef Search ADS PubMed  9 O’Neill LD, Wallstedt B, Erika B, Hartvigsen J: Factors associated with dropout in medical education: a literature review. Med Educ  2011; 45: 440– 54. Google Scholar CrossRef Search ADS PubMed  10 LaRochelle J, Denton G: Using pre-clerkship variables to identify high-risk students. In: Handbook On Medical Student Evaluation And Assessment , Ed 1, pp 49– 58. North Syracuse, Gegensatz Press, 2015. 11 Hemann BA, Durning SJ, Kelly WF, Dong T, Pangaro LN, Hemmer PA: Referral for competency committee review for poor performance on the internal medicine clerkship is associated with poor performance in internship. Mil Med  2015; 180( 4 Suppl): 71– 6. Google Scholar CrossRef Search ADS PubMed  Author notes The views expressed herein are those of the authors and not necessarily those of the Department of Defense or other federal agencies. Published by Oxford University Press on behalf of the Association of Military Surgeons of the United States 2018. This work is written by (a) US Government employee(s) and is in the public domain in the US. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Military Medicine Oxford University Press

A Longitudinal Study of Commonly Used Admissions Measures and Disenrollment from Medical School and Graduate Medical Education Probation or Termination from Training

Loading next page...
 
/lp/ou_press/a-longitudinal-study-of-commonly-used-admissions-measures-and-oNDjrKd7ag
Publisher
Oxford University Press
Copyright
Published by Oxford University Press on behalf of the Association of Military Surgeons of the United States 2018.
ISSN
0026-4075
eISSN
1930-613X
D.O.I.
10.1093/milmed/usy069
Publisher site
See Article on Publisher Site

Abstract

Abstract Introduction This is an empirical study to better understand commonly used medical school admission measures and disenrollment decisions during undergraduate medical education as well as graduate medical education (GME) probation or termination decisions. Materials and Methods Based on the data of USUHS medical students matriculating between 1998 and 2011 (N = 2,460), we compared medical school graduates and those disenrolled from medical school on MCAT scores, undergraduate BCPM (Biology, Chemistry, Physics, Math) GPA, and undergraduate overall GPA. We also reported more specific reasons for disenrollment decisions. Next, we compared the students who were referred to the student promotion committee (SPC) with other students on these measures. Moving onto GME, we compared trainees who were put on probation or terminated from training with those who were not on MCAT and undergraduate GPA measures. In addition, we examined the association between being referred to the SPC and GME probation or termination. Results There were 2,347 graduates and 113 disenrolled students from medical school (4.8%). For the disenrolled students, 43 (38.7%) students were disenrolled for exclusively (or primarily) non-academic reasons, and 68 (61.3%) were disenrolled for exclusively (or primarily) academic reasons. The t-tests showed statistically significant differences on the MCAT score of the first attempt (t(2,449) = 7.22, P < 0.01, Cohen’s d = 0.70), average MCAT score (t(2,449) = 4.22, P < 0.01, Cohen’s d = 0.41), and highest MCAT score (t(2,449) = 3.51, P < 0.01, Cohen’s d = 0.34). Logistic regression model selection also revealed that the best predictor for disenrollment was the first MCAT score (exp(b) = 0.83, 95% CI = (0.78, 0.88)). No significant differences on these measures were found from the group comparisons on SPC and GME probation or termination. There was no significant association between SPC appearance and GME probation or termination. Conclusions Academic difficulties, especially in the basic sciences, appear to be the most common factor for disenrollment from medical school. These students also had lower MCAT scores, particularly on the first attempt. The MCAT performance indicators and undergraduate GPA were consistently lower, but not statistically significant, for those who appeared before SPC or were put on probation or terminated from training during GME. INTRODUCTION Over the past 30 years, the graduation rate for medical students has decreased. According to a recent AAMC’s report,1 the 4-yr graduation rate of U.S. medical school students was 82.5% and the 5-yr graduation rate was 94.1%. Further, many scholars have argued that the United States is currently facing a physician shortage based on projected health care needs for society2,3 and undergraduate medical education (UME) disenrollment as well as graduate medical education (GME) probation or termination from training can contribute to the reduction in physician supply. Additionally, learners who are disenrolled from medical school or placed on probation or terminated from training during GME result in significant financial, time and other resource costs to both the institution and the learner. Thus, work is needed to better understand commonly used admissions measures and their association with disenrollment decisions at UME as well as GME probation or termination. Studies on disenrollment must rely on access to large amounts of longitudinal data, often from a single institution, given that UME disenrollment decisions and GME probation or termination are quite rare in the medical profession. Prior studies investigating the prediction of the attrition rates and/or other adverse actions in medical education have focused on either medical school or graduate medical education, but not both.4,5 The present study reports an investigation at the Uniformed Services University of the Health Sciences (USUHS). Upon matriculation, students are commissioned as officers in the Uniformed Services (Army, Navy, Air Force, and Public Health Service) and after graduation, the vast majority train in military-affiliated programs across the country. In this essentially “closed” system, the school can track the students’ performance from admission through all 4 years of medical school and into graduate medical education programs and beyond. A prior study of the attrition of matriculating USUHS medical students enrolled during the years 1980–1999 demonstrated that MCAT scores of disenrolled students (mean = 28.5) were significantly lower than that of those who graduated or were currently enrolled (mean = 31). In addition, the undergraduate GPA was slightly lower in the disenrolled group (mean = 3.41) as compared to the graduated or currently enrolled group (mean = 3.46), but the difference was not statistically significant.6 The purpose of the present study was to seek evidence to answer the research questions that centered around the comparisons of MCAT scores (score of the first attempt, score of the last attempt, average score of all attempts, the highest score of all attempts, and number of attempts) and undergraduate GPAs between medical school graduates and the disenrolled students as well as between GME trainees who were placed on probation or termination and those who were not. We used the MCAT and undergraduate GPAs in part because these are commonly used measures by all medical schools in the USA and thus may provide useful information for purposes of graduation rate questions. We investigated the following research questions. First, do disenrolled medical students differ from graduates on MCAT and undergraduate GPA? Second, what are the reasons for disenrollment from medical school? Third, do students who appeared before the SPC differ from the non-SPC students on MCAT performance or undergraduate GPA? Fourth, do the students who were put on probation or termination during internship, residency, and/or fellowship differ from those who did not on MCAT performance and undergraduate GPA? Lastly, is there an association between SPC appearance and being put on probation or termination during GME? METHODS Participants and measures USUHS is the only federal medical university in the USA. It matriculates approximately 170 students per year into the School of Medicine. The study cohort was medical students matriculating between 1998 and 2011 (N = 2,460). This is the cohort that we had full access to their MCAT scores, undergraduate GPAs, reasons of disenrollment, the Student Promotion Committee (SPC) indicator, and the GME probation or termination indicator. In this study, GME trainees include interns, residents, or fellows. For the MCAT scores, we collected or calculated five measures – score of the first attempt, score of the last attempt, average score of all attempts, the highest score of all attempts, and number of attempts. For the undergraduate GPAs, we collected data on both undergraduate BCPM (Biology, Chemistry, Physics, Math) GPA and the overall GPA. We collected the data on disenrollment as well as reasons for disenrollment which are maintained and updated by the Office of Student Affairs and the SPC. The research team reviewed the reasons for disenrollment and determined the categories and sub-categories of these reasons. The SPC is the official School of Medicine body that reviews matters of serious concern for violations of personal integrity and professional comportment in addition to poor academic performance. The School of Medicine charges the SPC with the responsibility for reviewing cognitive and non-cognitive performance. The committee consists of 10 faculty members appointed by the Dean of the School of Medicine, 2 faculty nominated by the faculty senate, and 2 students. Detailed information about the SPC, such as who were on the committee and what was the criterion of students being referred to SPC, can be found on USUHS’ website.7 We collected data from USUHS graduates who were trained at one of the GME programs administered by the National Capital Consortium (NCC) regarding probation or termination training outcomes. The NCC includes Walter Reed National Military Medical Center, Malcolm Grow Clinics and Surgical Center, and Fort Belvoir Community Hospital. Data Analyses We conducted t-tests to examine whether the mean differences of the various MCAT measures and undergraduate GPA measures were statistically significant between groups as described in the first, third, and fourth research questions. We applied two-tailed t-tests throughout. As the sample size in the graduates’ group was considerably larger than the disenrolled students group, we took a random smaller sample of the graduates group and reran the t-tests. In the results section, we only report the numbers calculated from the whole sample because the results of using a random smaller sample were aligned with the entire sample. Similarly, t-tests were conducted to investigate the differences between students who appeared before the SPC and other students, and those GME trainees who were put on probation or termination with other GME trainees. In addition, we performed contingency table analysis to examine whether appearing before the SPC was related to GME probation or termination to answer the last research question. In the Results section, we report the values of the inferential test statistics if they were statistically significant. Otherwise, we report only descriptive statistics. Results Medical School Graduates vs. Disenrolled Students: MCAT and Undergraduate GPA Comparison In the study cohort, there were 2,347 medical school graduates and 113 (4.8%) disenrolled students from medical school. The comparison of the means of the MCAT scores between the graduates and the disenrolled students is shown in Figure 1. The t-tests using the entire sample showed statistical differences on the MCAT score of the first attempt (graduates = 29.43, disenrolled students = 27.04, t(2,449) = 7.22, P < 0.01, Cohen’s d = 0.70), the average MCAT score (graduates = 29.14, disenrolled students = 27.79, t(2449) = 4.22, P < 0.01, Cohen’s d = 0.41), and the highest MCAT score (graduates = 29.78, disenrolled students = 28.73, t(2,449) = 3.51, P < 0.01, Cohen’s d = 0.34). When we compared the number of MCAT attempts, 1,449 (63.9%) of the graduates and 56 (53.3%) of the disenrolled students made one attempt only; 770 (32.8%) graduates and 42 (40.0%) of the disenrolled students made two attempts. The t-tests on the undergraduate BCPM GPA (mean of graduates = 3.48, mean of disenrolled students = 3.44, t(2449) = 1.45, P > 0.05) and the overall GPA (mean of graduates = 3.52, mean of disenrolled students = 3.51, t(2,449) = 0.40, P > .05) did not reveal significant differences between the groups. FIGURE 1. View largeDownload slide The sample mean of the MCAT score measures of the graduates group and the entire disenrolled students group. Cohen’s d was added to comparisons that yielded significant results from the t-tests. FIGURE 1. View largeDownload slide The sample mean of the MCAT score measures of the graduates group and the entire disenrolled students group. Cohen’s d was added to comparisons that yielded significant results from the t-tests. Reasons of Disenrollment For the disenrolled students, 43 (38.7%) students were disenrolled for exclusively (or primarily) non-academic reasons, and 68 (61.3%) were disenrolled for exclusively (or primarily) academic reasons. We were able to identify the specific reasons for disenrollment for 76 students, among which were academic issues (55) and non-academic issues (21). The more detailed breakdown of disenrollment reasons is displayed in Table I. If we focus on those students disenrolled for academic reasons, the means were 26.95 for the first attempt, 28.15 for the average score, 29.39 for the last attempt, and 29.39 for the highest MCAT attempt. The means of the BCPM undergraduate GPA and the overall undergraduate GPA were 3.41 and 3.48. The comparison with the graduates yielded significant results on the first MCAT score (t(2,406) = 4.74, P < 0.01, Cohen’s d = 0.68) and the average MCAT score (t(2,406) = 4.74, P < 0.05, Cohen’s d = 0.31). Table I. Summary of Reasons of Disenrollment from Medical School (N = 76) Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Table I. Summary of Reasons of Disenrollment from Medical School (N = 76) Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Disenrollment Reasons  Frequency  Percentage  Academic issues  55  72.3   Basic sciences difficulty  41  53.9   Clinical sciences difficulty  14  18.4  Non-academic issues  21  27.7   Professionalism issue  4  5.3   Medical problem  1  1.3   Resigned (no academic or professionalism difficulties)  16  21.1  Students Referred to SPC vs. Others: MCAT and Undergraduate GPA Comparison Among the medical school graduates, 319 students (13.6%) appeared before the SPC and 2,028 (86.4%) did not. The sample means of the MCAT score measures and undergraduate GPA measures of the SPC group were consistently lower than the no SPC group but the differences were not statistically significant (see Table II). Table II. The Means and Standard Deviations (in Parenthesis) of the MCAT Measures and Undergraduate GPA Measures of the Students Who Were Recommended to SPC vs. Those Were not Recommended and Trainees Who Were Placed on Probation or Termination during GME vs. Those Who Were Not   MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)    MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)  Table II. The Means and Standard Deviations (in Parenthesis) of the MCAT Measures and Undergraduate GPA Measures of the Students Who Were Recommended to SPC vs. Those Were not Recommended and Trainees Who Were Placed on Probation or Termination during GME vs. Those Who Were Not   MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)    MCAT Score of the First Attempt  MCAT Score of the Last Attempt  Average MCAT Score  Highest MCAT Score  Undergraduate BCPM GPA  Undergraduate Overall GPA  SPC  28.94 (3.39)  28.26 (3.69)  28.56 (3.27)  29.43 (3.02)  3.43 (0.31)  3.46 (0.28)  Non-SPC  29.51 (3.28)  28.95 (3.56)  29.23 (3.21)  29.83 (3.02)  3.48 (0.31)  3.53 (0.31)  GME probation or termination  28.95 (3.97)  28.32 (3.42)  28.71 (3.36)  29.16 (3.44)  3.35 (0.32)  3.47 (0.24)  No GME probation or termination  29.65 (3.27)  29.03 (3.66)  29.40 (3.24)  29.87 (3.05)  3.43 (0.33)  3.52 (0.31)  Trainees Put on Probation or Termination vs. Others: MCAT and Undergraduate GPA Comparison Within the study cohort, 495 graduates entered graduate medical education at the NCC. Among these, we identified 26 trainees (5.3%) who were put on probation or termination during GME. The sample means and standard deviations of the MCAT score measures and BCPM comparison are also displayed in Table II. The differences between the groups (SPC vs. non-SPC and GME probation or termination vs. no GME probation or termination) were not statistically significant. However, the sample means of the MCAT score measures and undergraduate GPA measures were consistently lower than for those trainees who were put on probation or termination during GME. Logistic Regression Model Selection and Fit We conducted logistic regression model selection for the outcomes of disenrollment and GME probation or termination. From the explanatory variables of MCAT scores and undergraduate GPAs, the model selection results indicated that the best predictor for graduates vs. disenrolled students was the first MCAT score (exp(b) = 0.83, 95% CI = (0.78, 0.88)). With every one point increase in the first MCAT score, the odds of getting disenrolled decreased by 17%. We did not observe any significant predictor for the measure of trainees being put on probation or termination from training. Association Between SPC Measure and GME Probation/Termination The cross tabulation between the SPC and GME probation or termination indicated 375 with no SPC and no GME probation or termination, 20 with no SPC but GME probation or termination, 52 with SPC but no GME probation or termination, and 6 with both SPC and GME probation or termination. The risk of GME probation or termination was about two times greater for the SPC group than the no SPC group (the absolute risk of GME probation or termination of the SPC group = 0.10, the absolute risk of GME probation or termination of the non-SPC group = 0.05). The chi-square test of independence did not show significant association between these two measures (X2(1) = 0.2.61; P = 0.11). DISCUSSION Based on 14 years of data, we conducted a study to find evidence-based answers to several research questions that are relevant to both UME and GME. In terms of our first research question, disenrolled medical students showed consistently lower MCAT scores and undergraduate GPAs (both BCPM GPA and the overall GPA) than the graduates although the comparison on GPAs did not reach statistical significance. The difference was most obvious in the first attempt and the average MCAT scores. Our data suggest that these two MCAT measures may be the most important MCAT variables to consider if the goal is to minimize disenrollment rates. This finding was aligned with a previous study by Zhao et al. in which the authors concluded that the average MCAT score is the best measure when predicting USMLE Step 1 score from repeated MCAT scores.8 Given the range of MCAT scores for both disenrolled and non-disenrolled students, an MCAT threshold score to avoid disenrollment could not be determined. Returning to our second research question, the majority (72.3%) of the students were disenrolled for academic reasons. This is consistent with a systematic review of medical school drop outs that found that academic issues were most strongly associated with dropping out of medical school and, by contrast, there was no consistent pattern of demographic variables that predicted drop-out.9 Only about 5% of the students in our study were disenrolled because of non-academic reasons, including one or more professionalism issues, a medical problem, or resignation. It seems that academic challenges, either from basic sciences or clinical studies, should be medical educators’ primary focus when it comes to reducing the disenrollment rate of medical school. Comparing basic sciences and clinical sciences, we found that the former accounted for considerably higher (53.9% in this study) percentage of disenrollment than the latter (18.4%). Our study also suggested that efforts on helping academically challenged students get through the basic sciences education period seem to be the most important factor to consider in terms of reducing disenrollment. Research on identifying and providing remediation for students who struggled in the pre-clerkship years have been reported.10 Further, it should be noted that in this study approximately 20% of the 76 drop outs of medical school resigned for unknown reasons. This group should be further investigated in future studies. We also asked if students who appeared before the SPC differ from the non- SPC students on MCAT performance or undergraduate GPA. MCAT performance indicators and undergraduate GPA were lower for the students who appeared before SPC and the trainees who were put on probation or termination during GME than those who did not appear before the SPC or were put on probation or termination during GME, respectively, but none of these comparisons were statistically significant. This is consistent with the notion that clinical performance is only partly dependent upon medical knowledge. Similarly, preventing GME probation or termination from training was less clear. Comparing GME trainees who were put on probation or termination with those who were not did not yield statistically significant results. Notably, we did not observe significant associations between appearing before the SPC in medical school and GME probation or termination. These data suggest that commonly used measurements in medical school may not predict GME probation or termination. Our past research found that other indicators such as unsuccessful clinical performance which required remediation during the internal medicine clerkship is strongly associated with poor performance at the internship level.11 Our study had several limitations. First, our findings were based on the sample from a single federal medical school. The findings may not be generalizable to other civilian institutions. Second, we focused on the MCAT scores and undergraduate GPA in this study and the MCAT data used were the “old” MCAT and not reflective of the “new” MCAT scoring system that was changed in 2015. Further, we were not able to collect the GME probation or termination data of all the USU graduates – our findings were based on our medical school graduates who entered GME at the NCC, which may not be representative of our graduates as a whole. Additionally, in the present study we did not explore the specific reasons of SPC referral or GME probation or termination. Lastly, we can only infer association rather than causation and we did not explore the impact of remediation on disenrollment. Strengths of our study included the large cohort over many years and a relatively complete database. Additionally, our work spanned from medical school into graduate medical education. CONCLUSION This large-sample longitudinal study across undergraduate medical education and graduate medical education revealed that academic difficulties appear to be the most common factor for students dropping out of medical school. These students typically have lower MCAT scores, particularly the first MCAT attempt score. The students who appeared before the SPC during UME or had GME probation or termination had consistently lower MCAT score and undergraduate GPA than those who did not, but these comparisons were not statistically significant. Thus, basic science knowledge, especially as measured by performance on the Medical College Admissions Test (MCAT), should be recognized as an important component in the process of reducing the attrition of medical students. References 1 Association of American Medical Colleges: Graduation rates and attrition factors in U.S. medical school students. Anal Brief  2014; 14( 5): 1– 2. https://www.aamc.org/download/379220/data/may2014aib-graduationratesandattritionfactorsforusmedschools.pdf. 2 Kirch DG, Petelle K: Addressing the physician shortage: the peril of ignoring demography. JAMA  2017; 317( 19): 1947– 8. doi:10.1001/jama.2017.2714. Google Scholar CrossRef Search ADS PubMed  3 The complexities of physician supply and demand: Projections from 2013 to 2025 (Final Report). Prepared for Association of American Medical Colleges. Available at https://www.aamc.org/download/426242/data/ihsreportdownload.pdf; accessed August 1, 2017. 4 Caulfield M, Redden G, Sondheimer H: Graduation rates and attrition factors for U.S. medical school students. Anal Brief  2014; 14( 5). https://www.aamc.org/download/379220/data/may2014aib-graduationratesandattritionfactorsforusmedschools.pdf. 5 Durning SJ, Dong T, Hemmer PA, et al.  : Are commonly used premedical school or medical school measures associated with Board certification? Mil Med  2015; 180( 4 Suppl): 18– 23. Google Scholar CrossRef Search ADS PubMed  6 Stetto JE, Gackstetter GD, Cruess DF, Hooper TI: Variables associated with attrition from Uniformed Services University of the Health Sciences Medical School. Mil Med  2004; 2: 102– 7. Google Scholar CrossRef Search ADS   7 Uniformed Services University: 2018. Usuhsedu. Available at: https://www.usuhs.edu/students/campuslife/stuaffairsresources; accessed February 16, 2018. 8 Zhao X, Oppler S, Dunleavy D, Kroopnick M: Validity of four approaches of using repeaters' MCAT scores in medical school admissions to predict USMLE step 1 total scores. Acad Med  2010; 85: S64– 7. doi:10.1097/acm.0b013e3181ed38fc. Google Scholar CrossRef Search ADS PubMed  9 O’Neill LD, Wallstedt B, Erika B, Hartvigsen J: Factors associated with dropout in medical education: a literature review. Med Educ  2011; 45: 440– 54. Google Scholar CrossRef Search ADS PubMed  10 LaRochelle J, Denton G: Using pre-clerkship variables to identify high-risk students. In: Handbook On Medical Student Evaluation And Assessment , Ed 1, pp 49– 58. North Syracuse, Gegensatz Press, 2015. 11 Hemann BA, Durning SJ, Kelly WF, Dong T, Pangaro LN, Hemmer PA: Referral for competency committee review for poor performance on the internal medicine clerkship is associated with poor performance in internship. Mil Med  2015; 180( 4 Suppl): 71– 6. Google Scholar CrossRef Search ADS PubMed  Author notes The views expressed herein are those of the authors and not necessarily those of the Department of Defense or other federal agencies. Published by Oxford University Press on behalf of the Association of Military Surgeons of the United States 2018. This work is written by (a) US Government employee(s) and is in the public domain in the US.

Journal

Military MedicineOxford University Press

Published: Apr 27, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off