ADVANCING ASSESSMENT AND SELF-ASSESSMENT
The State of Evaluation in Internal Medicine Residency
Saima I. Chaudhry, MD, MSHS
, Eric Holmboe, MD
, and Brent W. Beasley, MD
Department of Medicine and Office of Graduate Medical Education, North Shore University Hospital, Manhasset, NY, USA;
of Internal Medicine, Philadelphia, PA, USA;
University of Missouri-Kansas City, Kansas City, MO, USA;
Office of Graduate Medical Education,
North Shore University Hospital, Manhasset, NY, USA.
BACKGROUND: There are no nationwide data on the
methods residency programs are using to assess trainee
competence. The Accreditation Council for Graduate
Medical Education (ACGME) has recommended tools that
programs can use to evaluate their trainees. It is unknown
if programs are adhering to these recommendations.
OBJECTIVE: To describe evaluation methods used by
our nation’s internal medicine residency programs and
assess adherence to ACGME methodological recommen-
dations for evaluation.
DESIGN: Nationwide survey.
PARTICIPANTS: All internal medicine programs regis-
tered with the Association of Program Directors of Internal
MEASUREMENTS: Descriptive statistics of programs
and tools used to evaluate competence; compliance with
ACGME recommended evaluative methods.
RESULTS: The response rate was 70%. Programs were
using an average of 4.2–6.0toolstoevaluatetheir
trainees with heavy reliance on rating forms. Direct
observation and practice and data-based tools were used
much less frequently. Most programs were using at least
1 of the Accreditation Council for Graduate Medical
Education (ACGME)’s “most desirable” methods of eval-
uation for all 6 measures of trainee competence. These
programs had higher support staff to resident ratios
than programs using less desirable evaluative methods.
CONCLUSIONS: Residency programs are using a large
number and variety of tools for evaluating the compe-
tence of their trainees. Most are complying with ACGME
recommended methods of evaluation especially if the
support staff to resident ratio is high.
KEY WORDS: graduate medical education; residency; ACGME;
J Gen Intern Med 23(7):1010–5
© Society of General Internal Medicine 2008
Effective trainee evaluation is a professional responsibility of
medical educators. In 1999, the Accreditation Council for
Graduate Medical Education (ACGME) endorsed the concept
of the core competencies and mandated programs to evaluate
their trainees in 6 broad areas: Patient Care, Medical Knowl-
edge, Professionalism, Communication, Practice-Based Learn-
ing, and Systems-Based Practice. In 2002, programs were
expected to start implementing competency evaluation. By
2006, the competencies were to be fully integrated into the
curriculum and evaluation of all trainees in the United States.
Although the Council has largely let programs decide how to
best assess each competency, they have provided recommenda-
tions to help programs tackle the important task of evaluation.
Such recommendations are included in the ACGME Out-
comes Project, and more specifically, in its Toolbox of Assessment
Methods, which provides information about the psychometric
qualities and feasibility of a variety of evaluative tools.
Several investigators have studied the use of these tools,
usually providing a description of a particular evaluation at a
single institution, and other times providing outcomes assess-
These studies provide programs with practical insight
regarding different evaluative methods. However, there has
been no nationwide description of the state of evaluation in
graduate medical education. Without such an analysis, we
cannot fully understand how programs determine the compe-
tence of their graduates or whether they are adhering to ACGME
methodological recommendations for evaluation.
The purpose of this study is to describe how competence is
evaluated in Internal Medicine graduate medical education in
the United States during the early stages of the evolution toward
outcomes-based education. We describe the frequency with
which various tools are used to assess trainees, along with the
number of tools used to measure each competency. We also
describe how well residency programs are complying with the
ACGME’s recommended evaluation methodologies. Finally, we
compare the characteristics of programs that use the ACGME’s
recommended evaluation methods for assessing all 6 compe-
tencies versus fewer than 6 competencies.
The Survey Task Force of the Association of Program Directors
in Internal Medicine (APDIM) developed a 74-item questionnaire
to obtain information about our nation’s residency programs.
Our main outcome included tools used to assess each of the 6
competencies. Our survey was comprised of 5 sections as-
sessing: 1) characteristics of the program including program
resources, 2) characteristics of faculty and staff, 3) character-
istics of residents, 4) characteristics of the program director,
and 5) tools used to assess each competency.