Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians

The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians BackgroundPublic disclosure of physician-specific performance data is becoming increasingly common. However, the influence that public reporting of outcome data has on the delivery of care by physicians who are being assessed is not well understood.MethodsSince 1994, the New York State Department of Health has collected and periodically published observed and risk-adjusted patient mortality rates for all interventional cardiologists practicing coronary angioplasty in the state. To assess the influence that these reports exert on the physicians being monitored, a questionnaire was administered in an anonymous manner to all interventional cardiologists included in the most recent report.ResultsThe vast majority (79%) of interventional cardiologists agreed or strongly agreed that the publication of mortality statistics has, in certain instances, influenced their decision regarding whether to perform angioplasty on individual patients. Physicians expressed an increased reluctance to intervene in critically ill patients with high expected mortality rates. Among the respondents, 83% agreed or strongly agreed that patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician-specific patients’ mortality rates. Additionally, 85% believed that the risk-adjustment model used in the Percutaneous Coronary Interventions (PCI) in New York State 1998-2000report is not sufficient to avoid punishing physicians who perform higher-risk interventions.ConclusionsPublic reporting of physician-specific outcome data may influence physicians to withhold procedures from patients at higher risk, even when physicians believe that the procedure might be beneficial. This phenomenon should be recognized in the design and administration of physician performance profiles.Public reporting of physician-specific outcome data are becoming more widespread. The New York State Department of Health, for example, currently collects and reports numbers of interventions and patient mortality statistics for all surgeons in the state who perform coronary artery bypass surgery and for all interventional cardiologists who perform coronary angioplasty.The principal objectives of these types of reports, often referred to as physician “scorecards,” are to allow the public to make better-informed decisions when choosing a physician or hospital, and to provide physicians and health care organizations with comparative data that ultimately will serve to improve the quality of health care.Although not well studied, several concerns have been raised regarding the impact that physician scorecards may have on patient care.Of primary concern, it has been suggested that physicians, knowing that their procedural mortality rates will be published, may be less inclined to offer procedures to patients at higher risk who, nevertheless, may benefit from undergoing the procedure.While most scorecards use risk-adjustment models in an attempt to account for differences in the severity of patients’ illnesses, physicians remain uncertain about the ability of these models to adequately credit practitioners who perform interventions on sicker patients. Thus, while scorecards provide the public with objective information, it remains uncertain whether these reports simultaneously alter the way that physicians care for patients.In an effort to better understand the influence that public reporting of mortality statistics has on the medical decision-making process of the individual physicians who are being scrutinized, we undertook the present study to evaluate systematically the opinions and experiences of all physicians who were included in the most recent Percutaneous Coronary Interventions (PCI) in New York State 1998-2000report(available at: http://www.health.state.ny.us/nysdoh/pci_1998-2000.pdf).METHODSSTUDY POPULATION AND QUESTIONNAIRE USEDThe study population consisted of all interventional cardiologists in New York State who were included in the most recent PCI in New York Statereport, PCI in New York State1998-2000, published in January 2003. This report contains physician-specific and hospital-specific outcome data for all percutaneous coronary revascularization procedures performed in New York State from January 1997 to December 2000. The report includes data on numbers of interventions performed as well as actual and risk-adjusted patient mortality rates for each physician.The study instrument consisted of a mail-based questionnaire designed to elicit the opinions of cardiologists regarding the PCI in New York Statereport and its potential impact on their clinical practice. Current mailing addresses for physicians were obtained from medical society membership directories, local telephone directories, and Internet-based directories.The questionnaire consisted of a single-page document that first asked each physician to provide selected background information (physician’s age, number of years since cardiology training was completed, board certification status in interventional cardiology, and annual number of angioplasty procedures; type of hospital where most of the physician’s procedures were performed; and risk-adjusted patient mortality rate as reported in the PCI in New York State1998-2000report). The questionnaire recipients were then presented with 9 statements or questions regarding the report and were asked to indicate whether they strongly agreed, agreed, disagreed, or strongly disagreed with each statement or question.To foster accuracy, we asked that the questionnaires be returned to us in an anonymous manner. The questionnaire was sent to all cardiologists twice (at a 4-week interval) in an attempt to maximize the response rate. The cardiologists who returned the questionnaire after the first mailing were instructed to disregard the second mailing.Based on an analysis of prior questionnaire-based studies published in the medical literature, our goal response rate for this study was 60%.STATISTICAL ANALYSISResponses were entered into a central database for analysis. Evaluation for potential relationships between background variables and responses to the various queries presented in the questionnaire was carried out with the &khgr;2test, with physician responses dichotomized (“disagree” and “strongly disagree” vs “agree” and “strongly agree”). P<.05 denoted statistical significance.RESULTSRESPONSE RATE AND PHYSICIAN VARIABLESQuestionnaires were mailed to the 186 interventional cardiologists included in the most recently published iteration of the PCI in New York Statereport, of whom 120 responded. This response rate of 65% met our preset goal. There were no “refusals,” ie, no individuals informed us that they refused to answer the questionnaire.Self-reported background information of the respondents is presented in Table 1. Seventy-eight percent of the respondents were board certified in interventional cardiology and there was a wide distribution among the physicians with respect to the number of years since cardiology training was completed, annual number of angioplasty procedures performed, and hospital type where the physicians performed most of their angioplasty procedures.Table 1. Self-Reported Background Information Provided by 120 Interventional CardiologistsPersonal Information*Cardiologists, %Board certified in interventional cardiology Yes78.3 No19.2 Not reported2.5Annual No. of coronary interventions performed ≤10015.0 101-20035.0 201-30027.5 >30021.7 Not reported0.8No. of years since completion of cardiology training ≤1030.8 11-1734.2 ≥1835.0Risk-adjusted patient mortality on last  PCI in NYSreport  0-0.536.7  0.51-1.024.2  1.01-1.50.8  >1.52.5 Not reported or “don’t know”25.9Hospital type where most interventions are performed University, teaching44.0 Community, teaching36.2 Community, nonteaching19.8Abbreviation: PCI in NYS, Percutaneous Coronary Interventions (PCI) in New York State 1998-2000.*Mean ± SD age was 47 ± 6.1 years.QUESTIONNAIRE RESPONSESResponses of the interventional cardiologists to the 9 specific queries related to the PCI in New York State1998-2000report are presented in Table 2. Three statements focused on the respondents’ overall perception of the report. Specifically, 76% of respondents disagreed or strongly disagreed that it serves to improve patient care in New York State. Likewise, 85% of the respondents disagreed or strongly disagreed that mortality statistics provide an accurate measure of physician quality, and 82% disagreed or strongly disagreed that mortality statistics provide information that is useful in selecting a physician or hospital for angioplasty.Table 2. Responses by 120 Interventional Cardiologists to a Survey QuestionnaireStatement or QuestionStrongly DisagreeDisagreeAgreeStrongly AgreeNo ResponseMortality statistics provide an accurate measure of physician quality38.346.712.50.81.7Mortality statistics provide information that is useful for the public in terms of selecting a physician or hospital for angioplasty32.549.215.01.71.7Public reporting of mortality statistics for interventional cardiologists should be adopted by other states25.845.024.24.20.8The Percutaneous Coronary Interventions (PCI) in New York State 1998-2000reportserves to improve patient care in New York State35.040.820.03.30.8Knowledge that mortality statistics will be publicly disseminated has, in certain instances, influenced your decision on whether to perform angioplasty on individual patients5.015.043.335.80.8Knowing that your patient mortality statistics will be made public influences your decision on whether to intervene in critically ill patients with high expected mortality rates (eg, patients with cardiogenic shock)6.712.531.747.51.7Patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician-specific mortality rates0.815.044.239.20.8The Percutaneous Coronary Interventions (PCI) in New York State 1998-2000reportuses a risk stratification model to derive a physician’s patient risk-adjusted mortality rate:  Do you agree or disagree that the model is sufficient to avoid penalizing   physicians who perform higher-risk interventions?52.532.510.03.31.7Physicians may report higher-risk conditions to improve their risk-adjusted mortality statistics2.58.355.033.30.8The overwhelming majority of respondents indicated that the presence of the PCI in New York Statereport influences their clinical decision-making process. Among the respondents, 83% agreed or strongly agreed that “patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician specific mortality rates.” Similarly, 79% agreed or strongly agreed that the knowledge that mortality statistics will be made public has, in certain instances, influenced their decision on whether to perform angioplasty on individual patients. Additionally, 79% agreed or strongly agreed that their decision-making process regarding whether to intervene in critically ill patients with high expected mortality rates is influenced by the presence of the scorecard.Physicians also indicated discomfort with the risk adjustment model used in the PCI in New York State1998-2000report. Among respondents, 85% disagreed or strongly disagreed with the statement that the risk stratification model used in the report is sufficient so as not to penalize physicians who perform higher-risk interventions. The vast majority (88%) of respondents agreed or strongly agreed that when submitting patient data for the PCI in New York Statereport, physicians may report higher-risk conditions for their patients&apos; features to improve their risk-adjusted mortality statistics.ASSOCIATION ANALYSISAnalysis was performed to determine whether significant associations existed between physician-specific variables (Table 1) and responses to the survey questions. Opinions regarding the PCI in New York Statereport were not associated with the respondent’s board certification status, annual numbers of angioplasty procedures, or patient risk-adjusted mortality rate on the most recent report. Younger physicians were significantly less likely than older physicians to agree that “the PCI in New York Statereport serves to improve patient care in New York State” (&khgr;22 = 8.2, P<.02), and that “public reporting of mortality statistics for interventional cardiologists should be adopted by other states” (&khgr;22 = 10.3, P<.01). Additionally, physicians who performed most of their angioplasty procedures at major university teaching hospitals were significantly more likely than other physicians to agree that “the publication of mortality statistics factors into their decision on whether to intervene in critically ill patients with high expected mortality rates” (&khgr;22 = 8.2, P<.02).COMMENTTo our knowledge, the present study represents the first systematic analysis of the attitudes and experiences of cardiologists with respect to the public reporting of physician-specific outcome data related to coronary angioplasty. The findings provide evidence that physician scorecards are not passive devices simply monitoring physician quality, but, rather, that they exert an active influence on the clinical decision-making process of the physicians who are being assessed. The overwhelming majority of New York State interventional cardiologists indicated that knowing that their patient mortality statistics were to be publicly reported has at times influenced their decision on whether to perform angioplasty on individual patients. Specifically, physicians indicated that the reporting of mortality statistics may prevent them from performing procedures on patients at higher risk even when they believe that these patients might benefit from the procedure. While the PCI in New York Statereport incorporates a risk-adjustment model, physicians did not believe that the methods of risk adjustment were sufficient to prevent penalizing cardiologists who perform higher-risk interventions. Opinions regarding the angioplasty scorecard were independent of most background variables, including board certification status, number of angioplasty procedures performed, or the respondent’s risk adjusted patient mortality rate.DO SCORECARDS IMPROVE PATIENT OUTCOMES?Reporting of hospital- and physician-specific outcome data has been undertaken for a variety of medical and surgical therapies.One fundamental aim of such scorecards is to promote improvements in quality of care, but documenting quality of care on outcomes has been difficult. Following initiation of the report on coronary artery bypass surgery in New York State in 1989, Hannan et alreported that the risk-adjusted mortality rate for bypass surgery in the state of New York declined by 41%. Other authors, however, have speculated that this improvement may have been attributable to factors unrelated to better surgical results, including an increased refusal by surgeons to operate on sicker patients; increased reporting of characteristics of patients at high risk in an attempt to reduce risk-adjusted mortality; improvements in surgical techniques that have been occurring across the United States; and increased numbers of out-of-state referrals for bypass surgery.As other states have begun to follow New York’s lead and report physician-specific outcome data for cardiac surgery, whether patients have been helped or harmed by these devices remains debated. While Hannan et alreported that in-hospital and 30-day risk-adjusted mortality rates following bypass surgery were significantly lower in regions with public dissemination programs than in the remainder of the United States, a provocative analysis by Dranove et alsuggested that such reductions in surgical mortality merely reflect a shift by surgeons in scorecard states toward operating on patients with lower illness severity. In comparing trends in New York State and Pennsylvania before and after these states adopted reporting systems for cardiac surgery, the authors noted that sicker patients became less likely to receive bypass surgery, and that mortality rates among higher-risk coronary patients have consequently increased as these individuals were less likely to be treated surgically. Indeed, in a survey of all cardiac surgeons practicing in New York State in 1997, 67% of respondents indicated that they had refused to offer bypass surgery to at least 1 patient within the previous year, with the New York State cardiac surgery reporting system “being an integral part of the decision-making process.”DO SCORECARDS INFLUENCE REFERRAL PATTERNS?The influence that published scorecard data has had on the behavior of health care consumers and referring physicians has likewise been a matter of debate. Among a cohort of 474 patients who underwent bypass surgery in Pennsylvania following the publication of that state’s Consumer Guide to Coronary Artery Bypass Surgery 1997-1999, less than 3% indicated that the scorecard had a significant impact on their choice of surgeon.While Hannan et alreported that 38% of surveyed cardiologists in New York State responded that the state’s bypass surgery report had affected their referrals to surgeons, another study found that 87% of cardiologists in Pennsylvania indicated that the Consumer Guide to Coronary Artery Bypass Surgeryhad minimal or no influence on their referral recommendations.SCORECARDS AND MEDICAL DECISION MAKINGSeveral investigators have raised the concern that practitioners may refuse to perform potentially beneficial procedures on sicker patients for fear that their reported mortality statistics be impacted adversely. In Pennsylvania, 59% of cardiologists reported increased difficulty in finding a surgeon willing to perform cardiac surgery in severely ill patients in need of such surgery after initiation of the state’s cardiac surgery reporting system, and 63% of cardiac surgeons reported that they were less or much less willing to operate on the most severely ill patients.Similarly, in our study, the vast majority of interventional cardiologists in New York State indicated that public reporting of mortality statistics influences their decision on whether to intervene in critically ill patients with high expected mortality rates. This is an important public health concern, as reluctance among physicians to perform coronary revascularization procedures on patients with appropriate indications may portend worse clinical outcomes. In 1 analysis, patients deemed to have an appropriate indication for coronary revascularization by an expert panel, but who did not undergo the procedure, were more likely to experience adverse clinical events than patients with similar indications who underwent revascularization.RISK ADJUSTMENTScorecards typically incorporate risk-adjustment models designed to account for differences in baseline risk among patients. The choice of specific details used in the design of any risk-adjustment model, such as the clinical variables to be included and the weighting of these variables, exerts a fundamental impact on the ultimate conclusions about the data set to which it applied.Some investigators have cautioned that, for coronary angioplasty, statistical modeling to discriminate between provider outcomes is limited by factors such as a low incidence of major procedural complications and the inclusion of surgeons who perform few procedures, whereas others have demonstrated excellent accuracy for such models.The Coronary Artery Bypass in New York State 1997-1999and the PCI in New York State 1998-2000reports use sophisticated models that have been validated and are periodically updated.Despite the care taken in constructing these models, Green and Wintfeldcalculated that predicted patient mortality rates assigned to surgeons in an early version of the Coronary Artery Bypass in New York Statereport explained only a small portion (7.3%) of the variance in the observed patient mortality among surgeons, and the power of the model to predict outcomes for individual patients was also low at 8.0%.Even if models adequately adjust risk for a population, physicians may remain skeptical about the ability of the model to adequately offset risk when confronted with a critically ill individual. Of the interventional cardiologists included in our survey, 85% believed that the risk stratification model used by New York State is insufficient to prevent penalizing physicians who perform higher-risk interventions. This is noteworthy, as patients with the greatest baseline risk often derive the greatest degree of benefit from more aggressive medical or surgical interventions. For example, in the Should We Emergently Revascularize Occluded Coronaries for Cardiogenic Shock (SHOCK) trial, which randomized patients with acute myocardial infarction complicated by cardiogenic shock to undergo early coronary revascularization (with bypass surgery or angioplasty) or initial medical stabilization, mortality rates were high (>50%) at 6 months in both treatment groups. Patients treated with a strategy of early revascularization, however, demonstrated a statistically significant 12.8% absolute reduction in death at 6 months compared with patients in whom early revascularization was not undertaken (50.3% vs 63.1%).One method by which physicians can lower their risk-adjusted patient mortality rates is to report conditions with higher risks than their patients’ actual conditions, a technique that has been referred to as “gaming.” While the New York State Department of Health attempts to verify the validity of the data that it receives by a variety of methods, 88% of the interventional cardiologists who responded to our survey believed that physicians still may use “gaming” to improve their risk-adjusted patient mortality. And while some care needs to be taken in the interpretation of this statement, as it asks the respondents to speculate on the actions of other cardiologists, the widespread belief that “gaming” exists points to a general lack of confidence in the accuracy of the risk-stratification algorithm used. The use of mortality as the sole outcome measure to indicate physician performance was also viewed critically, as 85% of our respondents expressed disagreement that mortality statistics provide an accurate measure of physician quality.While physician scorecards will likely continue to increase in number and scope, refinements should be considered to limit their potentially negative impact on physicians’ decisions. For example, the outcomes of patients with very high baseline risks (such as patients presenting with acute myocardial infarction and cardiogenic shock), who, nevertheless, have been shown to benefit from a particular intervention, could be considered separately from the “regular” outcomes on the report card. Additionally, physicians could benefit from better education regarding the risk-adjustment algorithms used in outcome reports.POTENTIAL LIMITATIONSThis study attempts to gain insights into the clinical decision-making process of physicians, which is difficult to measure by objective means. Our conclusions, as is the nature of all survey-based studies, assume that the subjective responses of the answering physicians reflect their true clinical thoughts and actions. It remains possible that preexisting biases toward the concept of “scorecarding” may have influenced the responses of some physicians. The response rate for our study was 65%, which exceeds the average response rate for survey-based studies published in the medical literature.It is possible that the views of the nonrespondents were different from those of the physicians who responded to the survey; however, it is unlikely that the qualitative conclusions of the study would have changed if additional responses had been received. Because the questionnaire was administered in an anonymous manner, we are not able to compare demographic data among responders and nonresponders.CONCLUSIONSThe results of this study indicate that public reporting of physician-specific outcomes influences physician behavior. At one extreme, physicians may withhold procedures from patients at higher risk for adverse outcomes, even when they believe that these procedures might be beneficial. In this respect, physician scorecards, designed as a method to improve health care outcomes for the general population, may paradoxically serve to adversely affect health care decisions for individual patients.This study also underscores the current uncertainty of whether physician scorecards are an accurate measure of physician quality, and if they serve to improve or harm patient outcomes. The prevalence and impact of scorecards are expected to continue to increase, as third-party payers have expressed interest in using these statistical models to ration payment. It is critical, therefore, that the impact of scorecards on physician decisions and patient outcomes be continually reassessed as these instruments evolve.Correspondence:Craig R. Narins, MD, Cardiology Unit, University of Rochester Medical Center, Box 679, Rochester, NY 14642 (craig_narins@urmc.rochester.edu).Accepted for Publication: July 14, 2004.REFERENCESNew York State Department of HealthCoronary Artery Bypass Surgery in New York State 1997-1999.Available at: http://www.health.state.ny.us/nysdoh/reports/pci_1998-2000.pdf. Accessed Novemeber 4, 2002New York State Department of HealthPercutaneous Coronary Interventions (PCI) in New York State 1998-2000.Albany: New York Dept of Health; January 2003MRChassinELHannanBADeBuonoBenefits and hazards of reporting medical outcomes publicly.N Engl J Med19963343943988538714EJTopolRMCaliffScorecard cardiovascular medicine. Its impact and future directions.Ann Intern Med199412065708250458ECSchneiderAMEpsteinInfluence of cardiac-surgery performance reports on referral practices and access to care: a survey of cardiovascular specialists.N Engl J Med19963352512568657242DDranoveDKesslerMMcClellanMSatterthwaiteIs more information better? the effects of “report cards” on health care providers.J Polit Econ2003111555588JHBurackPImpellizzeriPHomelJNCunninghamJrPublic reporting of surgical mortality: a survey of New York State cardiothoracic surgeons.Ann Thorac Surg19996811951200DAAschMKJedrziewskiNAChristakisResponse rates to mail surveys published in medical journals.J Clin Epidemiol199750112911369368521MNMarshallPGShekelleSLeathermanRHBrookThe public release of performance data: what do we expect to gain? a review of the evidence.JAMA20002831866187410770149ELHannanHKilburnJrMRaczEShieldsMRChassinImproving the outcomes of coronary artery bypass surgery in New York State.JAMA19942717617668114213NAOmoiguiDPMillerKJBrownOutmigration for coronary bypass surgery in an era of public dissemination of clinical outcomes.Circulation19969327338616936JGreenNWintfeldReport cards on cardiac surgeons: assessing New York State&apos;s approach.N Engl J Med1995332122912327700321WAGhaliASAshREHallMAMoskowitzStatewide quality improvement initiatives and mortality after cardiac surgery.JAMA19972773793829010169SFJencksCan large-scale interventions improve care?JAMA19972774194209010175ELHannanMSSarrazinDRDoranGERosenthalProvider profiling and quality improvement efforts in coronary artery bypass graft surgery: the effect on short-term mortality among Medicare beneficiaries.Med Care2003411164117214515112ECSchneiderAMEpsteinUse of public performance reports: a survey of patients undergoing cardiac surgery.JAMA1998279163816429613914ELHannanCCStoneTLBiddleBADeBuonoPublic release of cardiac surgery outcomes data in New York: what do New York State cardiologists think of it?Am Heart J1997134112011289424074HHemingwayAMCrookGFederUnderuse of coronary revascularization procedures in patients considered appropriate candidates for revascularization.N Engl J Med200134464565411228280RMPosesDKMcClishWRSmithResults of report cards for patients with congestive heart failure depend on the method used to adjust for severity.Ann Intern Med2000133102010877735LIIezzoniThe risks of risk adjustment.JAMA1997278160016079370507SGEllisNOmoiguiJABittlAnalysis and comparison of operator-specific outcomes in interventional cardiology: from a multicenter database of 4860 quality-controlled procedures.Circulation1996934314398565159MMoscucciGTO&apos;ConnorSGEllisValidation of risk adjustment models for in-hospital percutaneous transluminal coronary angioplasty mortality on an independent data set.J Am Coll Cardiol19993469269710483949DRHolmesFSelzerJMJohnstonModeling and risk prediction in the current era of interventional cardiology: a report from the National Heart, Lung, and Blood Institute Dynamic Registry.Circulation20031071871187612668511JSHochmanLASleeperJGWebbSHOCK InvestigatorsEarly revascularization in acute myocardial infarction complicated by cardiogenic shock: should we emergently revascularize occluded coronaries for cardiogenic shock.N Engl J Med199934162563410460813 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JAMA Internal Medicine American Medical Association

The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians

Loading next page...
 
/lp/american-medical-association/the-influence-of-public-reporting-of-outcome-data-on-medical-decision-MS0HrrH60h

References (10)

Publisher
American Medical Association
Copyright
Copyright 2005 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.
ISSN
2168-6106
eISSN
2168-6114
DOI
10.1001/archinte.165.1.83
pmid
15642879
Publisher site
See Article on Publisher Site

Abstract

BackgroundPublic disclosure of physician-specific performance data is becoming increasingly common. However, the influence that public reporting of outcome data has on the delivery of care by physicians who are being assessed is not well understood.MethodsSince 1994, the New York State Department of Health has collected and periodically published observed and risk-adjusted patient mortality rates for all interventional cardiologists practicing coronary angioplasty in the state. To assess the influence that these reports exert on the physicians being monitored, a questionnaire was administered in an anonymous manner to all interventional cardiologists included in the most recent report.ResultsThe vast majority (79%) of interventional cardiologists agreed or strongly agreed that the publication of mortality statistics has, in certain instances, influenced their decision regarding whether to perform angioplasty on individual patients. Physicians expressed an increased reluctance to intervene in critically ill patients with high expected mortality rates. Among the respondents, 83% agreed or strongly agreed that patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician-specific patients’ mortality rates. Additionally, 85% believed that the risk-adjustment model used in the Percutaneous Coronary Interventions (PCI) in New York State 1998-2000report is not sufficient to avoid punishing physicians who perform higher-risk interventions.ConclusionsPublic reporting of physician-specific outcome data may influence physicians to withhold procedures from patients at higher risk, even when physicians believe that the procedure might be beneficial. This phenomenon should be recognized in the design and administration of physician performance profiles.Public reporting of physician-specific outcome data are becoming more widespread. The New York State Department of Health, for example, currently collects and reports numbers of interventions and patient mortality statistics for all surgeons in the state who perform coronary artery bypass surgery and for all interventional cardiologists who perform coronary angioplasty.The principal objectives of these types of reports, often referred to as physician “scorecards,” are to allow the public to make better-informed decisions when choosing a physician or hospital, and to provide physicians and health care organizations with comparative data that ultimately will serve to improve the quality of health care.Although not well studied, several concerns have been raised regarding the impact that physician scorecards may have on patient care.Of primary concern, it has been suggested that physicians, knowing that their procedural mortality rates will be published, may be less inclined to offer procedures to patients at higher risk who, nevertheless, may benefit from undergoing the procedure.While most scorecards use risk-adjustment models in an attempt to account for differences in the severity of patients’ illnesses, physicians remain uncertain about the ability of these models to adequately credit practitioners who perform interventions on sicker patients. Thus, while scorecards provide the public with objective information, it remains uncertain whether these reports simultaneously alter the way that physicians care for patients.In an effort to better understand the influence that public reporting of mortality statistics has on the medical decision-making process of the individual physicians who are being scrutinized, we undertook the present study to evaluate systematically the opinions and experiences of all physicians who were included in the most recent Percutaneous Coronary Interventions (PCI) in New York State 1998-2000report(available at: http://www.health.state.ny.us/nysdoh/pci_1998-2000.pdf).METHODSSTUDY POPULATION AND QUESTIONNAIRE USEDThe study population consisted of all interventional cardiologists in New York State who were included in the most recent PCI in New York Statereport, PCI in New York State1998-2000, published in January 2003. This report contains physician-specific and hospital-specific outcome data for all percutaneous coronary revascularization procedures performed in New York State from January 1997 to December 2000. The report includes data on numbers of interventions performed as well as actual and risk-adjusted patient mortality rates for each physician.The study instrument consisted of a mail-based questionnaire designed to elicit the opinions of cardiologists regarding the PCI in New York Statereport and its potential impact on their clinical practice. Current mailing addresses for physicians were obtained from medical society membership directories, local telephone directories, and Internet-based directories.The questionnaire consisted of a single-page document that first asked each physician to provide selected background information (physician’s age, number of years since cardiology training was completed, board certification status in interventional cardiology, and annual number of angioplasty procedures; type of hospital where most of the physician’s procedures were performed; and risk-adjusted patient mortality rate as reported in the PCI in New York State1998-2000report). The questionnaire recipients were then presented with 9 statements or questions regarding the report and were asked to indicate whether they strongly agreed, agreed, disagreed, or strongly disagreed with each statement or question.To foster accuracy, we asked that the questionnaires be returned to us in an anonymous manner. The questionnaire was sent to all cardiologists twice (at a 4-week interval) in an attempt to maximize the response rate. The cardiologists who returned the questionnaire after the first mailing were instructed to disregard the second mailing.Based on an analysis of prior questionnaire-based studies published in the medical literature, our goal response rate for this study was 60%.STATISTICAL ANALYSISResponses were entered into a central database for analysis. Evaluation for potential relationships between background variables and responses to the various queries presented in the questionnaire was carried out with the &khgr;2test, with physician responses dichotomized (“disagree” and “strongly disagree” vs “agree” and “strongly agree”). P<.05 denoted statistical significance.RESULTSRESPONSE RATE AND PHYSICIAN VARIABLESQuestionnaires were mailed to the 186 interventional cardiologists included in the most recently published iteration of the PCI in New York Statereport, of whom 120 responded. This response rate of 65% met our preset goal. There were no “refusals,” ie, no individuals informed us that they refused to answer the questionnaire.Self-reported background information of the respondents is presented in Table 1. Seventy-eight percent of the respondents were board certified in interventional cardiology and there was a wide distribution among the physicians with respect to the number of years since cardiology training was completed, annual number of angioplasty procedures performed, and hospital type where the physicians performed most of their angioplasty procedures.Table 1. Self-Reported Background Information Provided by 120 Interventional CardiologistsPersonal Information*Cardiologists, %Board certified in interventional cardiology Yes78.3 No19.2 Not reported2.5Annual No. of coronary interventions performed ≤10015.0 101-20035.0 201-30027.5 >30021.7 Not reported0.8No. of years since completion of cardiology training ≤1030.8 11-1734.2 ≥1835.0Risk-adjusted patient mortality on last  PCI in NYSreport  0-0.536.7  0.51-1.024.2  1.01-1.50.8  >1.52.5 Not reported or “don’t know”25.9Hospital type where most interventions are performed University, teaching44.0 Community, teaching36.2 Community, nonteaching19.8Abbreviation: PCI in NYS, Percutaneous Coronary Interventions (PCI) in New York State 1998-2000.*Mean ± SD age was 47 ± 6.1 years.QUESTIONNAIRE RESPONSESResponses of the interventional cardiologists to the 9 specific queries related to the PCI in New York State1998-2000report are presented in Table 2. Three statements focused on the respondents’ overall perception of the report. Specifically, 76% of respondents disagreed or strongly disagreed that it serves to improve patient care in New York State. Likewise, 85% of the respondents disagreed or strongly disagreed that mortality statistics provide an accurate measure of physician quality, and 82% disagreed or strongly disagreed that mortality statistics provide information that is useful in selecting a physician or hospital for angioplasty.Table 2. Responses by 120 Interventional Cardiologists to a Survey QuestionnaireStatement or QuestionStrongly DisagreeDisagreeAgreeStrongly AgreeNo ResponseMortality statistics provide an accurate measure of physician quality38.346.712.50.81.7Mortality statistics provide information that is useful for the public in terms of selecting a physician or hospital for angioplasty32.549.215.01.71.7Public reporting of mortality statistics for interventional cardiologists should be adopted by other states25.845.024.24.20.8The Percutaneous Coronary Interventions (PCI) in New York State 1998-2000reportserves to improve patient care in New York State35.040.820.03.30.8Knowledge that mortality statistics will be publicly disseminated has, in certain instances, influenced your decision on whether to perform angioplasty on individual patients5.015.043.335.80.8Knowing that your patient mortality statistics will be made public influences your decision on whether to intervene in critically ill patients with high expected mortality rates (eg, patients with cardiogenic shock)6.712.531.747.51.7Patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician-specific mortality rates0.815.044.239.20.8The Percutaneous Coronary Interventions (PCI) in New York State 1998-2000reportuses a risk stratification model to derive a physician’s patient risk-adjusted mortality rate:  Do you agree or disagree that the model is sufficient to avoid penalizing   physicians who perform higher-risk interventions?52.532.510.03.31.7Physicians may report higher-risk conditions to improve their risk-adjusted mortality statistics2.58.355.033.30.8The overwhelming majority of respondents indicated that the presence of the PCI in New York Statereport influences their clinical decision-making process. Among the respondents, 83% agreed or strongly agreed that “patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician specific mortality rates.” Similarly, 79% agreed or strongly agreed that the knowledge that mortality statistics will be made public has, in certain instances, influenced their decision on whether to perform angioplasty on individual patients. Additionally, 79% agreed or strongly agreed that their decision-making process regarding whether to intervene in critically ill patients with high expected mortality rates is influenced by the presence of the scorecard.Physicians also indicated discomfort with the risk adjustment model used in the PCI in New York State1998-2000report. Among respondents, 85% disagreed or strongly disagreed with the statement that the risk stratification model used in the report is sufficient so as not to penalize physicians who perform higher-risk interventions. The vast majority (88%) of respondents agreed or strongly agreed that when submitting patient data for the PCI in New York Statereport, physicians may report higher-risk conditions for their patients&apos; features to improve their risk-adjusted mortality statistics.ASSOCIATION ANALYSISAnalysis was performed to determine whether significant associations existed between physician-specific variables (Table 1) and responses to the survey questions. Opinions regarding the PCI in New York Statereport were not associated with the respondent’s board certification status, annual numbers of angioplasty procedures, or patient risk-adjusted mortality rate on the most recent report. Younger physicians were significantly less likely than older physicians to agree that “the PCI in New York Statereport serves to improve patient care in New York State” (&khgr;22 = 8.2, P<.02), and that “public reporting of mortality statistics for interventional cardiologists should be adopted by other states” (&khgr;22 = 10.3, P<.01). Additionally, physicians who performed most of their angioplasty procedures at major university teaching hospitals were significantly more likely than other physicians to agree that “the publication of mortality statistics factors into their decision on whether to intervene in critically ill patients with high expected mortality rates” (&khgr;22 = 8.2, P<.02).COMMENTTo our knowledge, the present study represents the first systematic analysis of the attitudes and experiences of cardiologists with respect to the public reporting of physician-specific outcome data related to coronary angioplasty. The findings provide evidence that physician scorecards are not passive devices simply monitoring physician quality, but, rather, that they exert an active influence on the clinical decision-making process of the physicians who are being assessed. The overwhelming majority of New York State interventional cardiologists indicated that knowing that their patient mortality statistics were to be publicly reported has at times influenced their decision on whether to perform angioplasty on individual patients. Specifically, physicians indicated that the reporting of mortality statistics may prevent them from performing procedures on patients at higher risk even when they believe that these patients might benefit from the procedure. While the PCI in New York Statereport incorporates a risk-adjustment model, physicians did not believe that the methods of risk adjustment were sufficient to prevent penalizing cardiologists who perform higher-risk interventions. Opinions regarding the angioplasty scorecard were independent of most background variables, including board certification status, number of angioplasty procedures performed, or the respondent’s risk adjusted patient mortality rate.DO SCORECARDS IMPROVE PATIENT OUTCOMES?Reporting of hospital- and physician-specific outcome data has been undertaken for a variety of medical and surgical therapies.One fundamental aim of such scorecards is to promote improvements in quality of care, but documenting quality of care on outcomes has been difficult. Following initiation of the report on coronary artery bypass surgery in New York State in 1989, Hannan et alreported that the risk-adjusted mortality rate for bypass surgery in the state of New York declined by 41%. Other authors, however, have speculated that this improvement may have been attributable to factors unrelated to better surgical results, including an increased refusal by surgeons to operate on sicker patients; increased reporting of characteristics of patients at high risk in an attempt to reduce risk-adjusted mortality; improvements in surgical techniques that have been occurring across the United States; and increased numbers of out-of-state referrals for bypass surgery.As other states have begun to follow New York’s lead and report physician-specific outcome data for cardiac surgery, whether patients have been helped or harmed by these devices remains debated. While Hannan et alreported that in-hospital and 30-day risk-adjusted mortality rates following bypass surgery were significantly lower in regions with public dissemination programs than in the remainder of the United States, a provocative analysis by Dranove et alsuggested that such reductions in surgical mortality merely reflect a shift by surgeons in scorecard states toward operating on patients with lower illness severity. In comparing trends in New York State and Pennsylvania before and after these states adopted reporting systems for cardiac surgery, the authors noted that sicker patients became less likely to receive bypass surgery, and that mortality rates among higher-risk coronary patients have consequently increased as these individuals were less likely to be treated surgically. Indeed, in a survey of all cardiac surgeons practicing in New York State in 1997, 67% of respondents indicated that they had refused to offer bypass surgery to at least 1 patient within the previous year, with the New York State cardiac surgery reporting system “being an integral part of the decision-making process.”DO SCORECARDS INFLUENCE REFERRAL PATTERNS?The influence that published scorecard data has had on the behavior of health care consumers and referring physicians has likewise been a matter of debate. Among a cohort of 474 patients who underwent bypass surgery in Pennsylvania following the publication of that state’s Consumer Guide to Coronary Artery Bypass Surgery 1997-1999, less than 3% indicated that the scorecard had a significant impact on their choice of surgeon.While Hannan et alreported that 38% of surveyed cardiologists in New York State responded that the state’s bypass surgery report had affected their referrals to surgeons, another study found that 87% of cardiologists in Pennsylvania indicated that the Consumer Guide to Coronary Artery Bypass Surgeryhad minimal or no influence on their referral recommendations.SCORECARDS AND MEDICAL DECISION MAKINGSeveral investigators have raised the concern that practitioners may refuse to perform potentially beneficial procedures on sicker patients for fear that their reported mortality statistics be impacted adversely. In Pennsylvania, 59% of cardiologists reported increased difficulty in finding a surgeon willing to perform cardiac surgery in severely ill patients in need of such surgery after initiation of the state’s cardiac surgery reporting system, and 63% of cardiac surgeons reported that they were less or much less willing to operate on the most severely ill patients.Similarly, in our study, the vast majority of interventional cardiologists in New York State indicated that public reporting of mortality statistics influences their decision on whether to intervene in critically ill patients with high expected mortality rates. This is an important public health concern, as reluctance among physicians to perform coronary revascularization procedures on patients with appropriate indications may portend worse clinical outcomes. In 1 analysis, patients deemed to have an appropriate indication for coronary revascularization by an expert panel, but who did not undergo the procedure, were more likely to experience adverse clinical events than patients with similar indications who underwent revascularization.RISK ADJUSTMENTScorecards typically incorporate risk-adjustment models designed to account for differences in baseline risk among patients. The choice of specific details used in the design of any risk-adjustment model, such as the clinical variables to be included and the weighting of these variables, exerts a fundamental impact on the ultimate conclusions about the data set to which it applied.Some investigators have cautioned that, for coronary angioplasty, statistical modeling to discriminate between provider outcomes is limited by factors such as a low incidence of major procedural complications and the inclusion of surgeons who perform few procedures, whereas others have demonstrated excellent accuracy for such models.The Coronary Artery Bypass in New York State 1997-1999and the PCI in New York State 1998-2000reports use sophisticated models that have been validated and are periodically updated.Despite the care taken in constructing these models, Green and Wintfeldcalculated that predicted patient mortality rates assigned to surgeons in an early version of the Coronary Artery Bypass in New York Statereport explained only a small portion (7.3%) of the variance in the observed patient mortality among surgeons, and the power of the model to predict outcomes for individual patients was also low at 8.0%.Even if models adequately adjust risk for a population, physicians may remain skeptical about the ability of the model to adequately offset risk when confronted with a critically ill individual. Of the interventional cardiologists included in our survey, 85% believed that the risk stratification model used by New York State is insufficient to prevent penalizing physicians who perform higher-risk interventions. This is noteworthy, as patients with the greatest baseline risk often derive the greatest degree of benefit from more aggressive medical or surgical interventions. For example, in the Should We Emergently Revascularize Occluded Coronaries for Cardiogenic Shock (SHOCK) trial, which randomized patients with acute myocardial infarction complicated by cardiogenic shock to undergo early coronary revascularization (with bypass surgery or angioplasty) or initial medical stabilization, mortality rates were high (>50%) at 6 months in both treatment groups. Patients treated with a strategy of early revascularization, however, demonstrated a statistically significant 12.8% absolute reduction in death at 6 months compared with patients in whom early revascularization was not undertaken (50.3% vs 63.1%).One method by which physicians can lower their risk-adjusted patient mortality rates is to report conditions with higher risks than their patients’ actual conditions, a technique that has been referred to as “gaming.” While the New York State Department of Health attempts to verify the validity of the data that it receives by a variety of methods, 88% of the interventional cardiologists who responded to our survey believed that physicians still may use “gaming” to improve their risk-adjusted patient mortality. And while some care needs to be taken in the interpretation of this statement, as it asks the respondents to speculate on the actions of other cardiologists, the widespread belief that “gaming” exists points to a general lack of confidence in the accuracy of the risk-stratification algorithm used. The use of mortality as the sole outcome measure to indicate physician performance was also viewed critically, as 85% of our respondents expressed disagreement that mortality statistics provide an accurate measure of physician quality.While physician scorecards will likely continue to increase in number and scope, refinements should be considered to limit their potentially negative impact on physicians’ decisions. For example, the outcomes of patients with very high baseline risks (such as patients presenting with acute myocardial infarction and cardiogenic shock), who, nevertheless, have been shown to benefit from a particular intervention, could be considered separately from the “regular” outcomes on the report card. Additionally, physicians could benefit from better education regarding the risk-adjustment algorithms used in outcome reports.POTENTIAL LIMITATIONSThis study attempts to gain insights into the clinical decision-making process of physicians, which is difficult to measure by objective means. Our conclusions, as is the nature of all survey-based studies, assume that the subjective responses of the answering physicians reflect their true clinical thoughts and actions. It remains possible that preexisting biases toward the concept of “scorecarding” may have influenced the responses of some physicians. The response rate for our study was 65%, which exceeds the average response rate for survey-based studies published in the medical literature.It is possible that the views of the nonrespondents were different from those of the physicians who responded to the survey; however, it is unlikely that the qualitative conclusions of the study would have changed if additional responses had been received. Because the questionnaire was administered in an anonymous manner, we are not able to compare demographic data among responders and nonresponders.CONCLUSIONSThe results of this study indicate that public reporting of physician-specific outcomes influences physician behavior. At one extreme, physicians may withhold procedures from patients at higher risk for adverse outcomes, even when they believe that these procedures might be beneficial. In this respect, physician scorecards, designed as a method to improve health care outcomes for the general population, may paradoxically serve to adversely affect health care decisions for individual patients.This study also underscores the current uncertainty of whether physician scorecards are an accurate measure of physician quality, and if they serve to improve or harm patient outcomes. The prevalence and impact of scorecards are expected to continue to increase, as third-party payers have expressed interest in using these statistical models to ration payment. It is critical, therefore, that the impact of scorecards on physician decisions and patient outcomes be continually reassessed as these instruments evolve.Correspondence:Craig R. Narins, MD, Cardiology Unit, University of Rochester Medical Center, Box 679, Rochester, NY 14642 (craig_narins@urmc.rochester.edu).Accepted for Publication: July 14, 2004.REFERENCESNew York State Department of HealthCoronary Artery Bypass Surgery in New York State 1997-1999.Available at: http://www.health.state.ny.us/nysdoh/reports/pci_1998-2000.pdf. Accessed Novemeber 4, 2002New York State Department of HealthPercutaneous Coronary Interventions (PCI) in New York State 1998-2000.Albany: New York Dept of Health; January 2003MRChassinELHannanBADeBuonoBenefits and hazards of reporting medical outcomes publicly.N Engl J Med19963343943988538714EJTopolRMCaliffScorecard cardiovascular medicine. Its impact and future directions.Ann Intern Med199412065708250458ECSchneiderAMEpsteinInfluence of cardiac-surgery performance reports on referral practices and access to care: a survey of cardiovascular specialists.N Engl J Med19963352512568657242DDranoveDKesslerMMcClellanMSatterthwaiteIs more information better? the effects of “report cards” on health care providers.J Polit Econ2003111555588JHBurackPImpellizzeriPHomelJNCunninghamJrPublic reporting of surgical mortality: a survey of New York State cardiothoracic surgeons.Ann Thorac Surg19996811951200DAAschMKJedrziewskiNAChristakisResponse rates to mail surveys published in medical journals.J Clin Epidemiol199750112911369368521MNMarshallPGShekelleSLeathermanRHBrookThe public release of performance data: what do we expect to gain? a review of the evidence.JAMA20002831866187410770149ELHannanHKilburnJrMRaczEShieldsMRChassinImproving the outcomes of coronary artery bypass surgery in New York State.JAMA19942717617668114213NAOmoiguiDPMillerKJBrownOutmigration for coronary bypass surgery in an era of public dissemination of clinical outcomes.Circulation19969327338616936JGreenNWintfeldReport cards on cardiac surgeons: assessing New York State&apos;s approach.N Engl J Med1995332122912327700321WAGhaliASAshREHallMAMoskowitzStatewide quality improvement initiatives and mortality after cardiac surgery.JAMA19972773793829010169SFJencksCan large-scale interventions improve care?JAMA19972774194209010175ELHannanMSSarrazinDRDoranGERosenthalProvider profiling and quality improvement efforts in coronary artery bypass graft surgery: the effect on short-term mortality among Medicare beneficiaries.Med Care2003411164117214515112ECSchneiderAMEpsteinUse of public performance reports: a survey of patients undergoing cardiac surgery.JAMA1998279163816429613914ELHannanCCStoneTLBiddleBADeBuonoPublic release of cardiac surgery outcomes data in New York: what do New York State cardiologists think of it?Am Heart J1997134112011289424074HHemingwayAMCrookGFederUnderuse of coronary revascularization procedures in patients considered appropriate candidates for revascularization.N Engl J Med200134464565411228280RMPosesDKMcClishWRSmithResults of report cards for patients with congestive heart failure depend on the method used to adjust for severity.Ann Intern Med2000133102010877735LIIezzoniThe risks of risk adjustment.JAMA1997278160016079370507SGEllisNOmoiguiJABittlAnalysis and comparison of operator-specific outcomes in interventional cardiology: from a multicenter database of 4860 quality-controlled procedures.Circulation1996934314398565159MMoscucciGTO&apos;ConnorSGEllisValidation of risk adjustment models for in-hospital percutaneous transluminal coronary angioplasty mortality on an independent data set.J Am Coll Cardiol19993469269710483949DRHolmesFSelzerJMJohnstonModeling and risk prediction in the current era of interventional cardiology: a report from the National Heart, Lung, and Blood Institute Dynamic Registry.Circulation20031071871187612668511JSHochmanLASleeperJGWebbSHOCK InvestigatorsEarly revascularization in acute myocardial infarction complicated by cardiogenic shock: should we emergently revascularize occluded coronaries for cardiogenic shock.N Engl J Med199934162563410460813

Journal

JAMA Internal MedicineAmerican Medical Association

Published: Jan 10, 2005

There are no references for this article.