ENHANCING QUALITY AND SAFETY
An Evaluation of the Veterans Health Administration’s Clinical
Reminders System: A National Survey of Generalists
Constance H. Fung, MD, MSHS
1,2
, Jerry S. Tsai, MS
1
, Armine Lulejian, MPH
1
,
Peter Glassman, MBBS, MSc
1,2
, Emily Patterson, PhD
3,4
, Brad N. Doebbeling, MD, MSc
5,6,7
,
and Steven M. Asch, MD, MPH
1,2
1
VA Greater Los Angeles Healthcare System, Los Angeles, USA;
2
David Geffen School of Medicine at UCLA, Los Angeles, USA;
3
VA Getting at
Patient Safety (GAPS) Center, Cincinnati, USA;
4
Ohio State University, Columbus, USA;
5
VA Center of Excellence on Implementing Evidence-
based Practices, Richard L. Roudebush VA Medical Center, Indianapolis, USA;
6
Indiana University Center for Health Services and
Outcomes Research, Regenstrief Institute, Indianapolis, USA;
7
Department of Medicine, Indiana University School of Medicine, Indianapolis,
USA.
BACKGROUND: The Veterans Health Administration
(VHA) is a leader in developing computerized clinical
reminders (CCRs). Primary care physicians’ (PCPs)
evaluation of VHA CCRs could influence their future
development and use within and outside the VHA.
OBJECTIVE: Survey PCPs about usefulness and us-
ability of VHA CCRs.
DESIGN AND PARTICIPANTS: In a national survey,
VHA PCPs rated on a 7-point scale usefulness and
usability of VHA CCRs, and standardized scales (0–100)
were constructed. A hierarchical linear mixed (HLM)
model predicted physician- and facility-level variables
associated with more positive global assessment of
CCRs.
RESULTS: Four hundred sixty-one PCPs participated
(response rate, 69%). Scale Cronbach’s alpha ranged
from 0.62 to 0.82. Perceptions of VHA CCRs were pri-
marily in the midrange, where higher ratings indicate
more favorable attitudes (weighted standardized median,
IQR): global assessment (50, 28–61), clinical/situational
specificity (29, 17–42), integration with workflow/work-
load (39, 17–50), training (50, 33–67), VHA’s management
of CCR use (67, 50–83), design/interface (53, 40–67),
perceived role in CCR use (67, 50–83), and self-efficacy
(67, 57–78). In a HLM model, design/interface (p<.001),
self-efficacy (p<.001), integration with workflow/work-
load (p<.001), and training (p<.001) were associated with
more favorable global assessments of CCRs. Facilities in
the west as compared to the south (p=.033), and physi-
cians with academic affiliation (p=.045) had less favorable
global assessment of CCRs.
CONCLUSIONS: Our systematic assessment of end-
users’ perceptions of VHA CCRs suggests that CCRs
need to be developed and implemented with a continual
focus on improvement based on end-user feedback.
Potential target areas include better integration into the
primary care clinic workflow/workload.
KEY WORDS: decision support systems clinical; primary health care;
quality of health care; evidence-based medicine; medical records
systems computerized.
J Gen Intern Med 23(4):392–8
DOI: 10.1007/s11606-007-0417-8
© Society of General Internal Medicine 2007
INTRODUCTION
Randomized controlled trials have demonstrated that comput-
erized clinical reminders (CCRs) can improve adherence to
preventive and chronic disease guidelines.
1–6
CCRs, which are
triggered by information in specific data fields, are automated
decision support tools that can consistently prompt clinicians
to take evidence-based actions for prespecified types of condi-
tions. As such, CCRs, as part of a multicomponent approach
described in the Chronic Care Model,
7
are a promising quality
improvement technology to help implement evidence-based
care and improve care.
Despite the promise of CCRs, their effectiveness in practice is
mixed.
1,8
Understanding and characterizing clinicians’ percep-
tions of current CCR features and implementation strategies are
important steps toward improving CCR technology. A conceptual
framework developed for a study of Human Immunodeficiency
Virus (HIV) CCRs described several factors that influence
perceived usefulness and usability of CCRs.
9
These include
system, team, and individual factors as well as factors related to
the CCR interface design and clinical appropriateness.
The Veterans Health Administration (VHA), a leader in the
development of clinical reminders,
10,11
uses a variety of CCRs
to improve care.
12
Some VHA CCRs apply to all patients,
whereas others are triggered based upon diagnostic codes or
other information embedded in the electronic health record
(EHR). The types of CCRs available and the rules used to
determine eligibility, periodicity, and requirements for satisfy-
ing the CCRs closely mirror those found in the VHA’s External
Peer Review Program (EPRP),
12
a performance measurement
program. Although the EPRP does not require CCR use,
facilities implement CCRs to help clinicians meet EPRP targets.
However, a study of CCRs involving 451 clinicians in 8 VHA
facilities found considerable range in adherence to use of CCRs
(29–100%).
13
A recent study found widespread implementa-
JGIM
392