Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Funnel plots, performance variation and the Myocardial Infarction National Audit Project 2003–2004

Funnel plots, performance variation and the Myocardial Infarction National Audit Project 2003–2004 Background: Clinical governance requires health care professionals to improve standards of care and has resulted in comparison of clinical performance data. The Myocardial Infarction National Audit Project (a UK cardiology dataset) tabulates its performance. However funnel plots are the display method of choice for institutional comparison. We aimed to demonstrate that funnel plots may be derived from MINAP data and allow more meaningful interpretation of data. Methods: We examined the attainment of National Service Framework standards for all hospitals st st (n = 230) and all patients (n = 99,133) in the MINAP database between 1 April 2003 and 31 March 2004. We generated funnel plots (with control limits at 3 sigma) of Door to Needle and Call to Needle thrombolysis times, and the use of aspirin, beta-blockers and statins post myocardial infarction. Results: Only 87,427 patients fulfilled criteria for analysis of the use of secondary prevention drugs and 15,111 patients for analysis by Door to Needle and Call to Needle times (163 hospitals achieved the standards for Door to Needle times and 215 were within or above their control limits). One hundred and sixteen hospitals fell outside the 'within 25%' and 'more than 25%' standards for Call to Needle times, but 28 were below the lower control limits. Sixteen hospitals failed to reach the standards for aspirin usage post AMI and 24 remained below the lower control limits. Thirty hospitals were below the lower CL for beta-blocker usage and 49 outside the standard. Statin use was comparable. Conclusion: Funnel plots may be applied to a complex dataset and allow visual comparison of data derived from multiple health-care units. Variation is readily identified permitting units to appraise their practices so that effective quality improvement may take place. Background Kingdom (UK) Government [1]. It has been prompted by Improving the quality of care in the National Health Serv- incidents of failure of professional self-regulation, notably ice (NHS) by responding to variations in clinical processes the Bristol and Shipman cases [2,3] and resulted in the and outcomes is an imperative required by the United collection of comparative data at all levels of healthcare Page 1 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 provision. Though methods for using data to respond to be for these measures in real practice in the NHS. We also variation are not established, [4] funnel plots are sug- believe that most clinicians aim to be as good as their col- gested as the display method of choice for institutional leagues rather than seeking to meet a particular externally comparison [5]. set clinical target. The measures used are process rather than outcome measures. The three funnel plots for use of Funnel plots are based on Statistical Process Control secondary prevention medications show variation which (SPC), a set of methods for ongoing improvement of sys- seems related to volume of cases – larger volumes seem to tems, processes and outcomes [6-8]. Recently, compara- relate to lower achievement. Is there a systematic reason tive performance of UK cardiac surgeons has been why care, or recording of care, is more difficult in larger disseminated using these plots [9,10] and they could be units? Although the clinical processes measured do not used to study comparative performance measures in other vary solely by chance, comparing the variation in results datasets such as the Myocardial Infarction National Audit to chance helps us see where differences are important Project (MINAP) registry (a UK cardiology dataset that enough to investigate. We consider this is a good reason characteristically represents its results as performance for showing the results in a scatter as this variability is very tables) [11]. We aimed to demonstrate that funnel plots difficult to see in a table. may be derived from existing MINAP data and that they provide more meaningful interpretation of complex data. Special-cause and common-cause variation When the performance of clinical units is compared, one Methods might expect process measures to not necessarily display Database variability consistent with chance, as there are likely to be We studied all patients (and all hospitals in England who systematic reasons for differences. Variation may be attrib- manage acute myocardial infarction (AMI)) who were utable to either 'common-cause variation' or 'special- st entered into the MINAP database between 1 April 2003 cause variation'. We considered that units displayed 'spe- st and 31 March 2004. We tabulated the results of the cial cause variation' when their performance fell beyond MINAP database by the five variables reported in the the limit lines of the funnel plot and that they were MINAP Third Public Report [11], namely: Door to Needle located there due to the presence of systematic influences Time (DTN), Call to Needle Time (CTN), and the use of [12]. aspirin, beta-blockers and HMG-CoA reductase inhibitors for secondary prevention (that is, drugs that reduce the We considered that units displayed 'common-cause varia- risk of further AMIs). For the analysis we included all tion' when their performance fell within the limit lines of patients with an admission diagnosis of definite AMI that the funnel plot indicating that their performance varied had no justified delay to treatment and received thrombo- only by an amount consistent with random chance. It lytic treatment. (Justified delays to treatment included would be expected that the patient factors that influence hypertension, concern over risk of bleeding, delay in the clinical decisions being measured at unit level would obtaining consent, non-diagnostic initial electrocardio- be likely to present randomly to units across the UK and grams, cardiac arrest, or insufficient information). that decision-making might therefore be expected to vary by this amount if no systematic differences in the deci- Funnel plots sion-making processes and thresholds exists between For each target we generated scatter plots of performance, units. as a percentage, against the number of cases reported (the denominator for the percentage). The mean hospital per- Common-cause variation (because it is linked to chance), formance and exact binomial 3 sigma limits were calcu- is greatest when numbers of patients are small (left of fun- lated for all possible values for the number of cases and nel plot) and reduces as numbers of patients per unit used to create a funnel plot using the method described by increases. While being within the funnel plot's limits does Spiegelhalter [11]. MINAP set absolute targets for achieve- not exclude the possibility of more moderate or opposing ment and we made funnel charts using 3 sigma limits systematic influences being present, the most likely expla- around the target and around the mean. Only charts using nation for the variation seen for units within the limit a funnel based on the mean are presented (except for lines of the funnel plot is that it results from common- dtn30 for which both sets of limits are shown) as there cause variation. This is background noise that is a feature was no substantial difference between methods for of the process itself [13]. It may reflect local variances in thrombolysis measures and for the secondary medication hospital-specific practices and policies such as day-to-day measures relatively few hospitals fell within the funnel's variations in staffing levels or marginal differences in tran- limits and many fell above the upper limit when the limits sit times (for geographical reasons). were set around the target. The absolute targets are also arbitrary as no one knows what the precise levels should Page 2 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 Comparison tals were below the lower control limits. Sixteen hospitals We used the Planning and Performance Framework failed to reach the performance standards for aspirin (2003–6) standards to define achievement as those usage post AMI, but when analysed by SPC these 16 and a 'reaching the goal', 'within 25% of the goal' and 'more further 8 hospitals remained below the lower control than 25% from the goal'. Attainment of these standards is limit. For beta-blockers usage, 30 hospitals were below used in the derivation of performance tables for the the lower control limit and 49 outside the achievement annual MINAP reports. Achievement goals are different standards. HMG-CoA reductase inhibitor use was compa- for thrombolytic treatment (DTN and CTN) and second- rable from SPC analysis (31 below the lower control ary prevention (aspirin, beta-blockers and HMG-CoA limit) and performance standards (and 26 not achieving reductase inhibitors). For the DTN standard, 75% of the performance standard). patients were required to receive thrombolytic treatment within 30 minutes whereas for the CTN standard 48% of For the five MINAP output variables, between 26 and 43 patients were required to receive thrombolytic treatment hospitals where found to lie above their respective upper within 60 minutes in 2003–4. The standard for secondary control limits (table 1). prevention treatments was that 80% of patients dis- charged from hospital should receive aspirin, beta-block- Discussion ers and HMG-CoA reductase inhibitors. We used these Other statistical methods for health-care surveillance exist definitions to generate results for our dataset and to com- [14-16], but funnel plots offer readily interpretable plots pare them with the number of hospitals above, below or of multiple unit comparisons that allow for sample size, within the control limits as calculated by our funnel plots. using a scale that is intuitive for clinicians to use. Funnel The numbers of hospitals that had fewer than 20 cases, plots permit SPC assessment to be applied to a complex but met the MINAP analysis criteria were also recorded. dataset using crude (not case-mix adjusted) comparison of outcome data derived from multiple healthcare units. The analysis is not restricted by number of cases per unit. Results There were 99,133 patients in the MINAP database, which Common-cause and special-cause variation can be readily covered between 225 and 230 hospitals for the five varia- visualised (through the identification of units outside of bles. Only 87,427 fulfilled the inclusion criteria for analy- the funnel) and it permits each unit to appraise their local sis of the use of secondary prevention drugs (final practices e.g. low use of statins at discharge. diagnosis of AMI) and 15,111 for analysis by DTN and CTN times (as only ST-Elevation AMI patients are eligible Special-cause and common-cause variation for thrombolysis). For the five MINAP output variables Special-cause variation was identified in the MINAP fun- (CTN, DTN, use of aspirin, beta-blockers and HMG-CoA nel plots. The thrombolysis funnel plots revealed a wide reductase inhibitors) between 8 and 25 hospitals were dispersion of data suggesting a single consistent process of excluded from the MINAP performance tabulation, but care was not occurring across the hospitals, but that mul- were included in the SPC analysis. tiple processes were producing the measured outcome. This is not surprising as the call to door process (carried The funnel plots for thrombolytic treatment goals demon- out by ambulance services) is a different process to door strated a wide dispersion of process data and nearly as to needle (carried out by hospitals). Though the second- many hospitals were above the control limits as below ary prevention funnel plots conformed to the control lim- (figures 1 and 2). The funnel plots for secondary preven- its, outliers were readily identified below the lower tion showed a similar amount of dispersion beyond con- control limits suggesting under achievement. A number of trol limits (figures 3, 4 and 5). hospitals lay more than 3 sigma above the mean signify- ing either good clinical practice or favourable systematic For DTN, we identified 163 hospitals that achieved the biases in the collection and submission of data. Just as performance standards, but 215 hospitals were found areas for improvement can be identified in some units, within the funnel or above the upper control limit (table unit processes that have contributed to high attainment of 1). For CTN, 116 hospitals fell outside the 'within 25%' National targets can be used as examples of 'good practice' and 'more than 25%' performance standards, whereas that might be reproduced in other hospitals. Where the using SPC we only identified 28 below the lower control number of cases is smaller than expected (perhaps limit. because of unusually high excepting of patients from the measure), the relative position along the x axis compared When we compared the numbers of hospitals achieving with other similar units is useful, although MINAP use performance standards for the medication targets with codes rather than hospital names as identifiers. funnel plot depictions, we found that despite them being 'within 25%' of their attainment standards, many hospi- Page 3 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 Table 1: Comparison of the numbers of hospitals achieving NSF goals and the number inside and outside 3 sigma limits. DTN CTN Aspirin Beta-blocker Statin Analysis by attainment of NSF targets Reaching the goal 163 102 191 151 180 Within 25% of the goal 51 0 16 46 24 More than 25% from the goal 8 116 0 3 2 Less than 20 cases 8 12 20 25 20 Total 230 230 227 225 226 Analysis by funnel plot Above UCL 33 32 40 43 34 Within limits 182 170 187 152 164 Below LCL 15 28* 24 30 31 Total 230 230 227 225 226 Discrepancy between table and funnel plot assessment of performance Within target and below LCL 0 0 8 0 5 Within 25% of target but below LCL 8 28* 16 27 24 Beyond 25% of target and below LCL 7 0 0 3 2 Within target but above UCL 27 32 37 44 35 Total discrepancy 42 60 61 74 66 DTN = door to needle time, CTN = call to needle time, Statin = HMG-CoA reductase inhibitor, NSF = National Service Framework for Coronary Heart Disease. *The 'within 25% of target' did not apply to CTN 60. In total, 108 hospitals were above the 48% target and 122 below it, of which 28 were also below the LCL. We noted that for some hospitals, high numbers of ondary prevention is performed for all patients having patients were receiving secondary prevention medications AMI (whether ST segment elevation, non-ST elevation, but this did not correspond to the numbers of patients threatened AMI or unconfirmed AMI). Alternatively, receiving thrombolysis. This is because analysis for sec- some units may have focused resources on one target (e.g. by provision of a 24 hour thrombolysis-nurse service) in F 30 Figure 1 unnel plot for percentag minutes of arriving in ho e of spipatients thr tal (DTN 30 o)mbolysed within Funnel plot for percentage of patients thrombolysed within Funnel plot for 60 minutes of calling for Figure 2 percentage of help (CTN 60 patients thrombolysed within ) 30 minutes of arriving in hospital (DTN 30). Data abstracted Funnel plot for percentage of patients thrombolysed within from the MINAP Third Public Report.[11] DTN = door to 60 minutes of calling for help (CTN 60). Data abstracted needle time, LCL = lower control limit, UCL = upper control from the MINAP Third Public Report.[11] CTN = call to limit, target = National Service Framework for coronary needle time, LCL = lower control limit, UCL = upper control heart disease goal of 75%. Funnel plot with control limits for limit, target = National Service Framework for coronary both mean performance and target performance. In this case heart disease goal of 48%. Mean and target performance the number of hospitals identified is similar. coincide at 48%. Page 4 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 F secondary prevention medication Figure 3 unnel plot for percentage of patients r on discharge eceiving from hos aspirin as pital Funnel reductase inhibitor as secondar discharg Figure 5 pe from hospital lot for percentage of pay tients r preven etion medication on ceiving a HMG-CoA Funnel plot for percentage of patients receiving aspirin as Funnel plot for percentage of patients receiving a HMG-CoA secondary prevention medication on discharge from hospital. reductase inhibitor as secondary prevention medication on Data abstracted from the MINAP Third Public Report.[11] discharge from hospital. Data abstracted from the MINAP LCL = lower control limit, UCL = upper control limit, target Third Public Report.[11] Statin = HMG-CoA reductase inhib- = National Service Framework for coronary heart disease itor, LCL = lower control limit, UCL = upper control limit, goal of 80%, target 25% = more than 25% from the goal. target = National Service Framework for coronary heart dis- ease goal of 80%, target 25% = more than 25% from the goal. contrast with a second unit focussing on an alternate tar- get (e.g. by provision and adherence to an acute coronary may be because the recording of care, might be more dif- syndrome care pathway). Other explanations include ficult in larger hospitals and/or the denominators for local variations in case-mix as might result from alternate some may be incorrect (some hospitals may have definition of AMI and availability, use and types of cardiac included patients who wouldn't have qualified clinically enzyme markers [17]. for secondary prevention drugs). If the funnel plots were labelled with the hospitals' names, each could then check The funnel plots for use of secondary prevention medica- they have denominators of the size expected. tions show variation which is related to volume of cases (larger volumes correspond to lower achievement). This Performance tabulation As expected, there was a discrepancy in counts across the hospitals between SPC methodology and performance tabulation [18]. Up to a third of counts may be scored dif- ferently when SPC methodology is used in preference to performance tabulation. In part, this is due to the con- struction of control limits around mean performance rather than the absolute performance target. It also reflects the ability of SPC to include all hospitals in the analysis and to more precisely demonstrate evidence of perform- ance based on sample size. Small hospitals are subject to greater sampling error in estimates of performance and an observed failure to meet a performance standard may occur when appropriate procedures are being followed (if by chance the small numbers of patients being treated had an unusual profile). The lack of an upper limit in the per- F b from hos Figure 4 unnel plot for percentag locker as secondary preventi pital e of patients r on medicati eceiving on on di a beta scharge formance table contributes between half and two thirds of Funnel plot for percentage of patients receiving a beta the discrepancy. blocker as secondary prevention medication on discharge from hospital. Data abstracted from the MINAP Third Public Performance tables polarise results. They create a state of Report.[11] LCL = lower control limit, UCL = upper control 'sanctuary' in hospitals above the cut-off and a state of limit, target = National Service Framework for coronary 'alarm' in those below it. Special-cause variation may be heart disease goal of 80%, target 25% = more than 25% from the goal. overlooked because results are not considered in the con- text of control limits (that may be used as a guide to when Page 5 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 it is uneconomic to look for assignable causes). SPC ance. In the event that units believe the implications of the allows those hospitals with smaller numbers of patients to funnel plots to be clinically invalid, then attention might have a greater acceptable variance avoiding the need for rather focus on possible anomalies in the process of data- them to be excluded. Moreover, the use of categories of gathering and reporting at that unit as compared to oth- performance (e.g. the 'within 25% category') may lead to ers. inaccurate hospital performance estimates. They may overestimate performance because despite some hospitals Conclusion being 'within 25%' of their attainment goals, many were The Myocardial Infarction National Audit Project is at the found to be below the lower control limits. This occurs in forefront of the clinically-led registries, with the collection hospitals with a greater number of cases (figure 4 and 5) of consistent, comprehensive and accurate clinical infor- and highlights the fact that the use of pre-set targets (that mation about a specific group of patients. It represents the are independent of case volume) is problematic. most robustly collected data of its type and as such is a key pioneering project in which different methods of analysis Because performance tables overlook special-cause varia- can be tested. We have demonstrated that the extensive tion, they may misdirect attention to less important com- clinical output data may be visualised for direct compari- mon-cause variation. The secondary prevention son across multiple units using funnel plots. They avoid tabulation of the performance standards suggests near the worst of the polarisation of results associated with the universal compliance with the targets, which although 'ticks and crosses' in performance tables and allow the reassuring, may incline hospitals towards thinking no fur- identification of potential areas of achievement that war- ther action is needed. It may also mask excessive treat- rant further attention to permit continuing improvement ment of some patients (e.g. the elderly, for which the in the quality of care offered to patients. The routine pub- evidence base is less convincing). Funnel plots identify lication of funnel plots summarising cardiology perform- those hospitals that are under performing and visually ance attainments will encourage the understanding, suggest that improvements may still be made in ade- reflection, processes analysis and subsequent improve- quately performing hospitals. Multi-level modelling ments in health-care at all levels of policy, planning and would enable a unit's progress to be plotted over time and actual delivery. identify those hospitals that are 'coasting' at just above the target [19]. Although the clinical processes measured do Competing interests not vary solely by chance, comparing the variation in The author(s) declare that they have no competing inter- results to chance helps us see where differences are impor- ests. tant enough to investigate. We consider this is a good rea- son for showing the results in a scatter as this variability is Authors' contributions difficult to see in a table. CG researched and wrote the manuscript. AR performed the statistics and wrote the article. PB and AH reviewed, Limitations of SPC wrote and critically appraised the article. All authors read Like all performance measurement and analysis methods, and approved the final manuscript. SPC funnel charts may produce both false negative and positive results. Performing within limits does not guaran- Acknowledgements The extract from the MINAP database was provided by Dr J S Birkhead, tee that a unit may not be underperforming though this Clinical Director of MINAP, the national audit of myocardial infarction, may either be too slight to detect or masked by other fac- Clinical Effectiveness and Evaluation Unit, Royal College of Physicians, Lon- tors. Being outside a control limit is not always abnormal don. There were no sources of funding. (special cause) either. Of the 230 hospitals, 2 or 3 may be outside the 3 sigma (99.8%) limits by chance alone. References Despite such false negative and positive results, it is 1. Health D: The new NHS: modern, dependable. London: The important to remember that truly detecting a 'special Stationary Office, 1997 (Cm 3807). . 2. R B: Harold Shipman's clinical practice, 1974-1998. London: cause' encourages further investigation at the level of the Stationary Office, 2001. . individual hospital (and not missing this opportunity). 3. inquiry K: Report of the public inquiry into children's heart Similarly funnel plots help prevent investigation of an surgery at the Bristol Royal Infirmary. London: The Station- ary Office, 2001. . outcome resulting from a common-cause as if it were a 4. Johnson J: Making self regulation credible. Through bench- special-cause. marking, peer review, appraisal-and management. BMJ 1998, 316:1847-1848. 5. Spiegelhalter D: Funnel plots for institutional comparison. Improvement activity Quality & Safety in Health Care 2002, 11:390-391. Credibly using data to raise local awareness among health 6. Carey RG: Improving healthcare with control charts: Basic and advanced SPC methods and case studies. 2003 Quality care staff of the need for improvements in processes is Press, Milwaukee. 2004. vital when there is under-achievement in clinical perform- 7. Bolsin S: Was Bristol an outlier? Lancet 2001, 358:2084. Page 6 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 8. Colson M, Bolsin S: The use of statistical process control meth- ods in monitoring clinical performance. Int J Qual Health Care 2003, 15:445. 9. Bridgewater B: Mortality data in adult cardiac surgery for named surgeons: retrrospective examination of prospec- tively collected data on coronary artery surgery and aortic valve replacement. BMJ 2005, 330:506-510. 10. Bridgewater B, Grayson AD, Jackson M, Brooks N, Grotte GJ, Keenan DJ, Millner R, Fabri BM, Jones M: Surgeon specific mortal- ity in adult cardiac surgery: comparison between crude and risk stratified data. BMJ 2003, 327:13-17. 11. Physicians R: How the NHS manages heart attacks. Third pub- lic report of the Myocardial Infarction National Audit Project. Clinical Effectiveness and Evaluation Unit, London 2004. 12. Ishikawa K: Guide to Quality Control. Tokyo, Japan, Productiveity Press 1976. 13. WE D: Out of the Crisis. Cambridge, MA:MIT,1986. . 14. Aylin P, Alves B, Best N, Cook A, Elliott P, Evans SJ, Lawrence AE, Murray GD, Pollock J, Spiegelhalter D: Comparison of UK paedi- atric cardiac surgical performance by analysis of routinely collected data 1984-96: was Bristol an outlier? Lancet 2001, 358:181-187. 15. Goldstein H, Speigelhalter DJ: League tables and their limita- tions: statistical issues in comparison of institutional per- formance. J R Stat Soc 1996, 159:385-443. 16. Smith AFM, West M, Gordon K, Knapp MS, Trimble MG: Monitor- ing kidney transplant patients. Statastician 1983, 32:46-54. 17. Fox KAA, Birkhead J, Wilcox R, Knight C, Barth J: British Cardiac Society Working Group on the definition of myocardial inf- arction. Heart 2004, 90:603-609. 18. Adab P, Rouse AM, Mohammed MA, Marshall T: Performance league tables: the NHS deserves better. BMJ 2002, 324:95-98. 19. Tekkis PP, McCulloch P, Steger AC, Benjamin IS, Poloniecki JD: Mor- tality control charts for comparing performance of surgical units: validation study using hospital mortality data. BMJ 2003, 326:786-788. Pre-publication history The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2261/6/34/prepub Publish with Bio Med Central and every scientist can read your work free of charge "BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours — you keep the copyright BioMedcentral Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp Page 7 of 7 (page number not for citation purposes) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png BMC Cardiovascular Disorders Springer Journals

Funnel plots, performance variation and the Myocardial Infarction National Audit Project 2003–2004

Download PDF
7 pages

Loading next page...
 
/lp/springer-journals/funnel-plots-performance-variation-and-the-myocardial-infarction-Z0lvZYJ0PQ

References (23)

Publisher
Springer Journals
Copyright
Copyright © 2006 by Gale et al; licensee BioMed Central Ltd.
Subject
Medicine & Public Health; Cardiology; Cardiac Surgery; Angiology; Blood Transfusion Medicine; Internal Medicine; Medicine/Public Health, general
eISSN
1471-2261
DOI
10.1186/1471-2261-6-34
pmid
16884535
Publisher site
See Article on Publisher Site

Abstract

Background: Clinical governance requires health care professionals to improve standards of care and has resulted in comparison of clinical performance data. The Myocardial Infarction National Audit Project (a UK cardiology dataset) tabulates its performance. However funnel plots are the display method of choice for institutional comparison. We aimed to demonstrate that funnel plots may be derived from MINAP data and allow more meaningful interpretation of data. Methods: We examined the attainment of National Service Framework standards for all hospitals st st (n = 230) and all patients (n = 99,133) in the MINAP database between 1 April 2003 and 31 March 2004. We generated funnel plots (with control limits at 3 sigma) of Door to Needle and Call to Needle thrombolysis times, and the use of aspirin, beta-blockers and statins post myocardial infarction. Results: Only 87,427 patients fulfilled criteria for analysis of the use of secondary prevention drugs and 15,111 patients for analysis by Door to Needle and Call to Needle times (163 hospitals achieved the standards for Door to Needle times and 215 were within or above their control limits). One hundred and sixteen hospitals fell outside the 'within 25%' and 'more than 25%' standards for Call to Needle times, but 28 were below the lower control limits. Sixteen hospitals failed to reach the standards for aspirin usage post AMI and 24 remained below the lower control limits. Thirty hospitals were below the lower CL for beta-blocker usage and 49 outside the standard. Statin use was comparable. Conclusion: Funnel plots may be applied to a complex dataset and allow visual comparison of data derived from multiple health-care units. Variation is readily identified permitting units to appraise their practices so that effective quality improvement may take place. Background Kingdom (UK) Government [1]. It has been prompted by Improving the quality of care in the National Health Serv- incidents of failure of professional self-regulation, notably ice (NHS) by responding to variations in clinical processes the Bristol and Shipman cases [2,3] and resulted in the and outcomes is an imperative required by the United collection of comparative data at all levels of healthcare Page 1 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 provision. Though methods for using data to respond to be for these measures in real practice in the NHS. We also variation are not established, [4] funnel plots are sug- believe that most clinicians aim to be as good as their col- gested as the display method of choice for institutional leagues rather than seeking to meet a particular externally comparison [5]. set clinical target. The measures used are process rather than outcome measures. The three funnel plots for use of Funnel plots are based on Statistical Process Control secondary prevention medications show variation which (SPC), a set of methods for ongoing improvement of sys- seems related to volume of cases – larger volumes seem to tems, processes and outcomes [6-8]. Recently, compara- relate to lower achievement. Is there a systematic reason tive performance of UK cardiac surgeons has been why care, or recording of care, is more difficult in larger disseminated using these plots [9,10] and they could be units? Although the clinical processes measured do not used to study comparative performance measures in other vary solely by chance, comparing the variation in results datasets such as the Myocardial Infarction National Audit to chance helps us see where differences are important Project (MINAP) registry (a UK cardiology dataset that enough to investigate. We consider this is a good reason characteristically represents its results as performance for showing the results in a scatter as this variability is very tables) [11]. We aimed to demonstrate that funnel plots difficult to see in a table. may be derived from existing MINAP data and that they provide more meaningful interpretation of complex data. Special-cause and common-cause variation When the performance of clinical units is compared, one Methods might expect process measures to not necessarily display Database variability consistent with chance, as there are likely to be We studied all patients (and all hospitals in England who systematic reasons for differences. Variation may be attrib- manage acute myocardial infarction (AMI)) who were utable to either 'common-cause variation' or 'special- st entered into the MINAP database between 1 April 2003 cause variation'. We considered that units displayed 'spe- st and 31 March 2004. We tabulated the results of the cial cause variation' when their performance fell beyond MINAP database by the five variables reported in the the limit lines of the funnel plot and that they were MINAP Third Public Report [11], namely: Door to Needle located there due to the presence of systematic influences Time (DTN), Call to Needle Time (CTN), and the use of [12]. aspirin, beta-blockers and HMG-CoA reductase inhibitors for secondary prevention (that is, drugs that reduce the We considered that units displayed 'common-cause varia- risk of further AMIs). For the analysis we included all tion' when their performance fell within the limit lines of patients with an admission diagnosis of definite AMI that the funnel plot indicating that their performance varied had no justified delay to treatment and received thrombo- only by an amount consistent with random chance. It lytic treatment. (Justified delays to treatment included would be expected that the patient factors that influence hypertension, concern over risk of bleeding, delay in the clinical decisions being measured at unit level would obtaining consent, non-diagnostic initial electrocardio- be likely to present randomly to units across the UK and grams, cardiac arrest, or insufficient information). that decision-making might therefore be expected to vary by this amount if no systematic differences in the deci- Funnel plots sion-making processes and thresholds exists between For each target we generated scatter plots of performance, units. as a percentage, against the number of cases reported (the denominator for the percentage). The mean hospital per- Common-cause variation (because it is linked to chance), formance and exact binomial 3 sigma limits were calcu- is greatest when numbers of patients are small (left of fun- lated for all possible values for the number of cases and nel plot) and reduces as numbers of patients per unit used to create a funnel plot using the method described by increases. While being within the funnel plot's limits does Spiegelhalter [11]. MINAP set absolute targets for achieve- not exclude the possibility of more moderate or opposing ment and we made funnel charts using 3 sigma limits systematic influences being present, the most likely expla- around the target and around the mean. Only charts using nation for the variation seen for units within the limit a funnel based on the mean are presented (except for lines of the funnel plot is that it results from common- dtn30 for which both sets of limits are shown) as there cause variation. This is background noise that is a feature was no substantial difference between methods for of the process itself [13]. It may reflect local variances in thrombolysis measures and for the secondary medication hospital-specific practices and policies such as day-to-day measures relatively few hospitals fell within the funnel's variations in staffing levels or marginal differences in tran- limits and many fell above the upper limit when the limits sit times (for geographical reasons). were set around the target. The absolute targets are also arbitrary as no one knows what the precise levels should Page 2 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 Comparison tals were below the lower control limits. Sixteen hospitals We used the Planning and Performance Framework failed to reach the performance standards for aspirin (2003–6) standards to define achievement as those usage post AMI, but when analysed by SPC these 16 and a 'reaching the goal', 'within 25% of the goal' and 'more further 8 hospitals remained below the lower control than 25% from the goal'. Attainment of these standards is limit. For beta-blockers usage, 30 hospitals were below used in the derivation of performance tables for the the lower control limit and 49 outside the achievement annual MINAP reports. Achievement goals are different standards. HMG-CoA reductase inhibitor use was compa- for thrombolytic treatment (DTN and CTN) and second- rable from SPC analysis (31 below the lower control ary prevention (aspirin, beta-blockers and HMG-CoA limit) and performance standards (and 26 not achieving reductase inhibitors). For the DTN standard, 75% of the performance standard). patients were required to receive thrombolytic treatment within 30 minutes whereas for the CTN standard 48% of For the five MINAP output variables, between 26 and 43 patients were required to receive thrombolytic treatment hospitals where found to lie above their respective upper within 60 minutes in 2003–4. The standard for secondary control limits (table 1). prevention treatments was that 80% of patients dis- charged from hospital should receive aspirin, beta-block- Discussion ers and HMG-CoA reductase inhibitors. We used these Other statistical methods for health-care surveillance exist definitions to generate results for our dataset and to com- [14-16], but funnel plots offer readily interpretable plots pare them with the number of hospitals above, below or of multiple unit comparisons that allow for sample size, within the control limits as calculated by our funnel plots. using a scale that is intuitive for clinicians to use. Funnel The numbers of hospitals that had fewer than 20 cases, plots permit SPC assessment to be applied to a complex but met the MINAP analysis criteria were also recorded. dataset using crude (not case-mix adjusted) comparison of outcome data derived from multiple healthcare units. The analysis is not restricted by number of cases per unit. Results There were 99,133 patients in the MINAP database, which Common-cause and special-cause variation can be readily covered between 225 and 230 hospitals for the five varia- visualised (through the identification of units outside of bles. Only 87,427 fulfilled the inclusion criteria for analy- the funnel) and it permits each unit to appraise their local sis of the use of secondary prevention drugs (final practices e.g. low use of statins at discharge. diagnosis of AMI) and 15,111 for analysis by DTN and CTN times (as only ST-Elevation AMI patients are eligible Special-cause and common-cause variation for thrombolysis). For the five MINAP output variables Special-cause variation was identified in the MINAP fun- (CTN, DTN, use of aspirin, beta-blockers and HMG-CoA nel plots. The thrombolysis funnel plots revealed a wide reductase inhibitors) between 8 and 25 hospitals were dispersion of data suggesting a single consistent process of excluded from the MINAP performance tabulation, but care was not occurring across the hospitals, but that mul- were included in the SPC analysis. tiple processes were producing the measured outcome. This is not surprising as the call to door process (carried The funnel plots for thrombolytic treatment goals demon- out by ambulance services) is a different process to door strated a wide dispersion of process data and nearly as to needle (carried out by hospitals). Though the second- many hospitals were above the control limits as below ary prevention funnel plots conformed to the control lim- (figures 1 and 2). The funnel plots for secondary preven- its, outliers were readily identified below the lower tion showed a similar amount of dispersion beyond con- control limits suggesting under achievement. A number of trol limits (figures 3, 4 and 5). hospitals lay more than 3 sigma above the mean signify- ing either good clinical practice or favourable systematic For DTN, we identified 163 hospitals that achieved the biases in the collection and submission of data. Just as performance standards, but 215 hospitals were found areas for improvement can be identified in some units, within the funnel or above the upper control limit (table unit processes that have contributed to high attainment of 1). For CTN, 116 hospitals fell outside the 'within 25%' National targets can be used as examples of 'good practice' and 'more than 25%' performance standards, whereas that might be reproduced in other hospitals. Where the using SPC we only identified 28 below the lower control number of cases is smaller than expected (perhaps limit. because of unusually high excepting of patients from the measure), the relative position along the x axis compared When we compared the numbers of hospitals achieving with other similar units is useful, although MINAP use performance standards for the medication targets with codes rather than hospital names as identifiers. funnel plot depictions, we found that despite them being 'within 25%' of their attainment standards, many hospi- Page 3 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 Table 1: Comparison of the numbers of hospitals achieving NSF goals and the number inside and outside 3 sigma limits. DTN CTN Aspirin Beta-blocker Statin Analysis by attainment of NSF targets Reaching the goal 163 102 191 151 180 Within 25% of the goal 51 0 16 46 24 More than 25% from the goal 8 116 0 3 2 Less than 20 cases 8 12 20 25 20 Total 230 230 227 225 226 Analysis by funnel plot Above UCL 33 32 40 43 34 Within limits 182 170 187 152 164 Below LCL 15 28* 24 30 31 Total 230 230 227 225 226 Discrepancy between table and funnel plot assessment of performance Within target and below LCL 0 0 8 0 5 Within 25% of target but below LCL 8 28* 16 27 24 Beyond 25% of target and below LCL 7 0 0 3 2 Within target but above UCL 27 32 37 44 35 Total discrepancy 42 60 61 74 66 DTN = door to needle time, CTN = call to needle time, Statin = HMG-CoA reductase inhibitor, NSF = National Service Framework for Coronary Heart Disease. *The 'within 25% of target' did not apply to CTN 60. In total, 108 hospitals were above the 48% target and 122 below it, of which 28 were also below the LCL. We noted that for some hospitals, high numbers of ondary prevention is performed for all patients having patients were receiving secondary prevention medications AMI (whether ST segment elevation, non-ST elevation, but this did not correspond to the numbers of patients threatened AMI or unconfirmed AMI). Alternatively, receiving thrombolysis. This is because analysis for sec- some units may have focused resources on one target (e.g. by provision of a 24 hour thrombolysis-nurse service) in F 30 Figure 1 unnel plot for percentag minutes of arriving in ho e of spipatients thr tal (DTN 30 o)mbolysed within Funnel plot for percentage of patients thrombolysed within Funnel plot for 60 minutes of calling for Figure 2 percentage of help (CTN 60 patients thrombolysed within ) 30 minutes of arriving in hospital (DTN 30). Data abstracted Funnel plot for percentage of patients thrombolysed within from the MINAP Third Public Report.[11] DTN = door to 60 minutes of calling for help (CTN 60). Data abstracted needle time, LCL = lower control limit, UCL = upper control from the MINAP Third Public Report.[11] CTN = call to limit, target = National Service Framework for coronary needle time, LCL = lower control limit, UCL = upper control heart disease goal of 75%. Funnel plot with control limits for limit, target = National Service Framework for coronary both mean performance and target performance. In this case heart disease goal of 48%. Mean and target performance the number of hospitals identified is similar. coincide at 48%. Page 4 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 F secondary prevention medication Figure 3 unnel plot for percentage of patients r on discharge eceiving from hos aspirin as pital Funnel reductase inhibitor as secondar discharg Figure 5 pe from hospital lot for percentage of pay tients r preven etion medication on ceiving a HMG-CoA Funnel plot for percentage of patients receiving aspirin as Funnel plot for percentage of patients receiving a HMG-CoA secondary prevention medication on discharge from hospital. reductase inhibitor as secondary prevention medication on Data abstracted from the MINAP Third Public Report.[11] discharge from hospital. Data abstracted from the MINAP LCL = lower control limit, UCL = upper control limit, target Third Public Report.[11] Statin = HMG-CoA reductase inhib- = National Service Framework for coronary heart disease itor, LCL = lower control limit, UCL = upper control limit, goal of 80%, target 25% = more than 25% from the goal. target = National Service Framework for coronary heart dis- ease goal of 80%, target 25% = more than 25% from the goal. contrast with a second unit focussing on an alternate tar- get (e.g. by provision and adherence to an acute coronary may be because the recording of care, might be more dif- syndrome care pathway). Other explanations include ficult in larger hospitals and/or the denominators for local variations in case-mix as might result from alternate some may be incorrect (some hospitals may have definition of AMI and availability, use and types of cardiac included patients who wouldn't have qualified clinically enzyme markers [17]. for secondary prevention drugs). If the funnel plots were labelled with the hospitals' names, each could then check The funnel plots for use of secondary prevention medica- they have denominators of the size expected. tions show variation which is related to volume of cases (larger volumes correspond to lower achievement). This Performance tabulation As expected, there was a discrepancy in counts across the hospitals between SPC methodology and performance tabulation [18]. Up to a third of counts may be scored dif- ferently when SPC methodology is used in preference to performance tabulation. In part, this is due to the con- struction of control limits around mean performance rather than the absolute performance target. It also reflects the ability of SPC to include all hospitals in the analysis and to more precisely demonstrate evidence of perform- ance based on sample size. Small hospitals are subject to greater sampling error in estimates of performance and an observed failure to meet a performance standard may occur when appropriate procedures are being followed (if by chance the small numbers of patients being treated had an unusual profile). The lack of an upper limit in the per- F b from hos Figure 4 unnel plot for percentag locker as secondary preventi pital e of patients r on medicati eceiving on on di a beta scharge formance table contributes between half and two thirds of Funnel plot for percentage of patients receiving a beta the discrepancy. blocker as secondary prevention medication on discharge from hospital. Data abstracted from the MINAP Third Public Performance tables polarise results. They create a state of Report.[11] LCL = lower control limit, UCL = upper control 'sanctuary' in hospitals above the cut-off and a state of limit, target = National Service Framework for coronary 'alarm' in those below it. Special-cause variation may be heart disease goal of 80%, target 25% = more than 25% from the goal. overlooked because results are not considered in the con- text of control limits (that may be used as a guide to when Page 5 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 it is uneconomic to look for assignable causes). SPC ance. In the event that units believe the implications of the allows those hospitals with smaller numbers of patients to funnel plots to be clinically invalid, then attention might have a greater acceptable variance avoiding the need for rather focus on possible anomalies in the process of data- them to be excluded. Moreover, the use of categories of gathering and reporting at that unit as compared to oth- performance (e.g. the 'within 25% category') may lead to ers. inaccurate hospital performance estimates. They may overestimate performance because despite some hospitals Conclusion being 'within 25%' of their attainment goals, many were The Myocardial Infarction National Audit Project is at the found to be below the lower control limits. This occurs in forefront of the clinically-led registries, with the collection hospitals with a greater number of cases (figure 4 and 5) of consistent, comprehensive and accurate clinical infor- and highlights the fact that the use of pre-set targets (that mation about a specific group of patients. It represents the are independent of case volume) is problematic. most robustly collected data of its type and as such is a key pioneering project in which different methods of analysis Because performance tables overlook special-cause varia- can be tested. We have demonstrated that the extensive tion, they may misdirect attention to less important com- clinical output data may be visualised for direct compari- mon-cause variation. The secondary prevention son across multiple units using funnel plots. They avoid tabulation of the performance standards suggests near the worst of the polarisation of results associated with the universal compliance with the targets, which although 'ticks and crosses' in performance tables and allow the reassuring, may incline hospitals towards thinking no fur- identification of potential areas of achievement that war- ther action is needed. It may also mask excessive treat- rant further attention to permit continuing improvement ment of some patients (e.g. the elderly, for which the in the quality of care offered to patients. The routine pub- evidence base is less convincing). Funnel plots identify lication of funnel plots summarising cardiology perform- those hospitals that are under performing and visually ance attainments will encourage the understanding, suggest that improvements may still be made in ade- reflection, processes analysis and subsequent improve- quately performing hospitals. Multi-level modelling ments in health-care at all levels of policy, planning and would enable a unit's progress to be plotted over time and actual delivery. identify those hospitals that are 'coasting' at just above the target [19]. Although the clinical processes measured do Competing interests not vary solely by chance, comparing the variation in The author(s) declare that they have no competing inter- results to chance helps us see where differences are impor- ests. tant enough to investigate. We consider this is a good rea- son for showing the results in a scatter as this variability is Authors' contributions difficult to see in a table. CG researched and wrote the manuscript. AR performed the statistics and wrote the article. PB and AH reviewed, Limitations of SPC wrote and critically appraised the article. All authors read Like all performance measurement and analysis methods, and approved the final manuscript. SPC funnel charts may produce both false negative and positive results. Performing within limits does not guaran- Acknowledgements The extract from the MINAP database was provided by Dr J S Birkhead, tee that a unit may not be underperforming though this Clinical Director of MINAP, the national audit of myocardial infarction, may either be too slight to detect or masked by other fac- Clinical Effectiveness and Evaluation Unit, Royal College of Physicians, Lon- tors. Being outside a control limit is not always abnormal don. There were no sources of funding. (special cause) either. Of the 230 hospitals, 2 or 3 may be outside the 3 sigma (99.8%) limits by chance alone. References Despite such false negative and positive results, it is 1. Health D: The new NHS: modern, dependable. London: The important to remember that truly detecting a 'special Stationary Office, 1997 (Cm 3807). . 2. R B: Harold Shipman's clinical practice, 1974-1998. London: cause' encourages further investigation at the level of the Stationary Office, 2001. . individual hospital (and not missing this opportunity). 3. inquiry K: Report of the public inquiry into children's heart Similarly funnel plots help prevent investigation of an surgery at the Bristol Royal Infirmary. London: The Station- ary Office, 2001. . outcome resulting from a common-cause as if it were a 4. Johnson J: Making self regulation credible. Through bench- special-cause. marking, peer review, appraisal-and management. BMJ 1998, 316:1847-1848. 5. Spiegelhalter D: Funnel plots for institutional comparison. Improvement activity Quality & Safety in Health Care 2002, 11:390-391. Credibly using data to raise local awareness among health 6. Carey RG: Improving healthcare with control charts: Basic and advanced SPC methods and case studies. 2003 Quality care staff of the need for improvements in processes is Press, Milwaukee. 2004. vital when there is under-achievement in clinical perform- 7. Bolsin S: Was Bristol an outlier? Lancet 2001, 358:2084. Page 6 of 7 (page number not for citation purposes) BMC Cardiovascular Disorders 2006, 6:34 http://www.biomedcentral.com/1471-2261/6/34 8. Colson M, Bolsin S: The use of statistical process control meth- ods in monitoring clinical performance. Int J Qual Health Care 2003, 15:445. 9. Bridgewater B: Mortality data in adult cardiac surgery for named surgeons: retrrospective examination of prospec- tively collected data on coronary artery surgery and aortic valve replacement. BMJ 2005, 330:506-510. 10. Bridgewater B, Grayson AD, Jackson M, Brooks N, Grotte GJ, Keenan DJ, Millner R, Fabri BM, Jones M: Surgeon specific mortal- ity in adult cardiac surgery: comparison between crude and risk stratified data. BMJ 2003, 327:13-17. 11. Physicians R: How the NHS manages heart attacks. Third pub- lic report of the Myocardial Infarction National Audit Project. Clinical Effectiveness and Evaluation Unit, London 2004. 12. Ishikawa K: Guide to Quality Control. Tokyo, Japan, Productiveity Press 1976. 13. WE D: Out of the Crisis. Cambridge, MA:MIT,1986. . 14. Aylin P, Alves B, Best N, Cook A, Elliott P, Evans SJ, Lawrence AE, Murray GD, Pollock J, Spiegelhalter D: Comparison of UK paedi- atric cardiac surgical performance by analysis of routinely collected data 1984-96: was Bristol an outlier? Lancet 2001, 358:181-187. 15. Goldstein H, Speigelhalter DJ: League tables and their limita- tions: statistical issues in comparison of institutional per- formance. J R Stat Soc 1996, 159:385-443. 16. Smith AFM, West M, Gordon K, Knapp MS, Trimble MG: Monitor- ing kidney transplant patients. Statastician 1983, 32:46-54. 17. Fox KAA, Birkhead J, Wilcox R, Knight C, Barth J: British Cardiac Society Working Group on the definition of myocardial inf- arction. Heart 2004, 90:603-609. 18. Adab P, Rouse AM, Mohammed MA, Marshall T: Performance league tables: the NHS deserves better. BMJ 2002, 324:95-98. 19. Tekkis PP, McCulloch P, Steger AC, Benjamin IS, Poloniecki JD: Mor- tality control charts for comparing performance of surgical units: validation study using hospital mortality data. BMJ 2003, 326:786-788. Pre-publication history The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2261/6/34/prepub Publish with Bio Med Central and every scientist can read your work free of charge "BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours — you keep the copyright BioMedcentral Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.asp Page 7 of 7 (page number not for citation purposes)

Journal

BMC Cardiovascular Disorders – Springer Journals

Published: Aug 2, 2006

There are no references for this article.