TY - JOUR
AU1 - Ambroggio,, Lilliam
AU2 - Schondelmeyer,, Amanda
AU3 - Hoefgen,, Erik
AU4 - Brady,, Patrick
AU5 - Shaughnessy,, Erin
AB - Abstract Each quality improvement (QI) project has an implicit study design, although these designs are not discussed as commonly as they are in clinical research. Most QI projects fall under the quasi-experimental study category, in which observations are made before and after the implementation of an intervention(s). The simplest and most commonly used for QI studies is the pre-post design, in which observations are made before and after each intervention that was implemented over a specified period. More sophisticated designs for QI studies enable a study team to draw stronger inferences about the system that is being changed and the individual effects of the interventions that are implemented. In the final commentary in this QI series, we discuss these study designs and focus on the strengths and weaknesses of more sophisticated designs, including cluster randomized, stepped-wedge, and factorial designs. Many quality improvement (QI) projects use traditional improvement approaches, including a pre-post study design, and analyze their effect over time using a run or control chart. For example, if the aim of a project is to improve the reliability of hand-washing, the improvement team might decide to test interventions such as instituting a marketing campaign on hand-washing, auditing hand-washing data and providing feedback, and increasing the availability of hand sanitizer. Using run or control chart rules, as discussed previously in this series, the team could determine if special cause occurred, but such a finding would have limitations [1, 2]. First, the team would not know which intervention or combination of interventions was responsible for the improvement, especially if the interventions overlapped temporally. Second, controlling for confounders (eg, changes in patient population over time) would not be accomplished. Third, potential interactions between interventions (eg, 2 QI interventions that have a greater effect in combination than when implemented alone) could not be understood. The cost of these limitations is that we often do not know what interventions are most effective and under what considerations. This lack of understanding can result in substantial time and expense in implementing interventions that might not be effective. Our focus in this final article in the QI series is to discuss alternative study designs that are used in QI studies and that have been covered well in the infectious disease literature [1, 3, 4]. Cluster randomized trials, stepped-wedge designs, and factorial designs, when executed correctly, are unlike basic QI study designs in that they decrease potential confounding. As detailed more later, the main advantage of these more sophisticated designs is that they result in an enhanced understanding of the specific improvement intervention(s) that affected a significant change in the outcome. These study designs are particularly valuable to consider when the evidence behind the improvement interventions is weak or when the intervention is costly or associated with potential adverse effects. Because some of these studies might be more complicated and resource intensive to execute, it is recommended that improvement teams consult with colleagues experienced in advanced improvement, epidemiology, and/or biostatistics to determine feasibility and appropriateness of the design before beginning the study. QUASI-EXPERIMENTAL DESIGN OVERVIEW Most QI projects use a study design within the quasi-experimental category. Quasi-experimental designs indicate that the intervention is implemented at a specified time that is known by the study team (eg, even if the study team did not implement the intervention [eg, hospital-wide policy change]) and/or the QI study team implemented the intervention [3, 5, 6]. A simple example of a quasi-experimental study is one with a pre-post design in which the QI study team records the outcome before the intervention, implements the intervention, and subsequently records the outcome after intervention implementation. One specific example from the infectious diseases literature is the addition of levofloxacin to a hospital formulary to determine if it decreases ciprofloxacin-susceptible Escherichia coli [7]. Studies with a quasi-experimental design can also include a control group (eg, a different hospital unit, a different consult service, etc.), which allows for stronger inferences between the intervention and outcome of interest to be assessed [3, 5]. Including a control group might be important in studies of infectious diseases that are seasonal in nature (eg, influenza, pneumonia) or when another intervention was implemented outside of the QI study team’s control (eg, publication of national guidelines) [8, 9]. It is important to note that power for quasi-experimental study designs increases as more data points are collected before and after each intervention that is implemented [6]. Analyses of these studies range from creating visual displays of data over time (eg, run or control charts), as discussed in a previous article in this series [1], to using more complicated modeling (eg, interrupted time-series analysis) [6]. The following sections describe types of experimental and quasi-experimental study designs that are more sophisticated than the pre-post test design. CLUSTER RANDOMIZED TRIAL In a cluster randomized trial, the unit of observation is at the group rather than individual level. The interventions are distributed randomly, also at the group rather than the individual level. For instance, if a QI project is being conducted within a hospital setting, the group or unit of study could include the hospital unit, a specific physician specialty (eg, infectious diseases physicians versus surgeons), or a team of residents. By establishing and defining the unit to which the intervention will be applied, contamination or unintended spread of the intervention that might occur among the individuals within a cluster is not a concern. In contrast, patient-level randomization is often impossible in QI studies because of the challenges involved in containing contamination from physicians or nurses who would likely care for patients in both the intervention and control groups. In a cluster randomized trial, all the individuals within a cluster will receive the same intervention at the same time (Figure 1). The intervention(s) then can be distributed randomly between the clusters by using a variety of randomization tools, including a random-number table, a coin flip, or a random-number generator. Figure 1. View largeDownload slide Planning process for the execution of a cluster randomized trial. Figure 1. View largeDownload slide Planning process for the execution of a cluster randomized trial. Randomization aims to balance both measured (eg, patient age) and unmeasured (eg, crowding in the home environment) confounders between the clusters. Cluster randomized trials traditionally allow for only some of the clusters to receive the intervention(s). However, 1 type of cluster randomized trial that is gaining more popularity is the stepped-wedge design [10]. This design is unique in that it allows the QI team to implement the intervention(s) across all clusters in a sequential fashion (Table 1) [11]. The investigator can randomize the order in which the clusters receive the intervention(s). This design confers the following 4 advantages: (1) it enhances the researchers’ ability to determine the interaction between time and the intervention; (2) the effectiveness of the intervention(s) can be compared within each cluster and between clusters in each time period; (3) practical and logistical concerns regarding financial and physical resources are addressed because the resources can be better targeted in any given time period; and (4) recruitment of sites might be easier when each site is ensured of getting the intervention (versus a traditional cluster randomized trial, in which half the sites would be controls for a potentially promising intervention) (Table 2). Table 1. Stepped Wedge Design Schematic Cluster Intervention According to Time Period 1 2 3 4 1 No intervention Intervention 1 Intervention 2 Intervention 3 3 No intervention No intervention Intervention 1 Intervention 2 2 No intervention No intervention No intervention Intervention 1 Cluster Intervention According to Time Period 1 2 3 4 1 No intervention Intervention 1 Intervention 2 Intervention 3 3 No intervention No intervention Intervention 1 Intervention 2 2 No intervention No intervention No intervention Intervention 1 View Large Table 1. Stepped Wedge Design Schematic Cluster Intervention According to Time Period 1 2 3 4 1 No intervention Intervention 1 Intervention 2 Intervention 3 3 No intervention No intervention Intervention 1 Intervention 2 2 No intervention No intervention No intervention Intervention 1 Cluster Intervention According to Time Period 1 2 3 4 1 No intervention Intervention 1 Intervention 2 Intervention 3 3 No intervention No intervention Intervention 1 Intervention 2 2 No intervention No intervention No intervention Intervention 1 View Large Table 2. Advantages and Disadvantages of Study Designs for QI Study Design Advantages Disadvantages Pre-post Easy to execute; easy to analyze; design is strengthened by including control group Limited inference can be drawn from the association between intervention(s) and improvement in the system Cluster randomized trial Reduces confounding that can exist across the system; control group included in the design; easy to analyze Not receiving the interventions might not be acceptable to the different clusters Stepped wedge Might be more acceptable financially or ethically to the study team; seasonal/historical influences on the data can be assessed; every cluster acts as a control group during different periods of intervention Analysis can be complicated; might take longer to implement interventions (each intervention is implemented in the cluster in a sequential fashion) Factorial study Tests for interactions between interventions; tests multiple factors on an outcome over time Larger number of factors, so greater level of complexity Study Design Advantages Disadvantages Pre-post Easy to execute; easy to analyze; design is strengthened by including control group Limited inference can be drawn from the association between intervention(s) and improvement in the system Cluster randomized trial Reduces confounding that can exist across the system; control group included in the design; easy to analyze Not receiving the interventions might not be acceptable to the different clusters Stepped wedge Might be more acceptable financially or ethically to the study team; seasonal/historical influences on the data can be assessed; every cluster acts as a control group during different periods of intervention Analysis can be complicated; might take longer to implement interventions (each intervention is implemented in the cluster in a sequential fashion) Factorial study Tests for interactions between interventions; tests multiple factors on an outcome over time Larger number of factors, so greater level of complexity View Large Table 2. Advantages and Disadvantages of Study Designs for QI Study Design Advantages Disadvantages Pre-post Easy to execute; easy to analyze; design is strengthened by including control group Limited inference can be drawn from the association between intervention(s) and improvement in the system Cluster randomized trial Reduces confounding that can exist across the system; control group included in the design; easy to analyze Not receiving the interventions might not be acceptable to the different clusters Stepped wedge Might be more acceptable financially or ethically to the study team; seasonal/historical influences on the data can be assessed; every cluster acts as a control group during different periods of intervention Analysis can be complicated; might take longer to implement interventions (each intervention is implemented in the cluster in a sequential fashion) Factorial study Tests for interactions between interventions; tests multiple factors on an outcome over time Larger number of factors, so greater level of complexity Study Design Advantages Disadvantages Pre-post Easy to execute; easy to analyze; design is strengthened by including control group Limited inference can be drawn from the association between intervention(s) and improvement in the system Cluster randomized trial Reduces confounding that can exist across the system; control group included in the design; easy to analyze Not receiving the interventions might not be acceptable to the different clusters Stepped wedge Might be more acceptable financially or ethically to the study team; seasonal/historical influences on the data can be assessed; every cluster acts as a control group during different periods of intervention Analysis can be complicated; might take longer to implement interventions (each intervention is implemented in the cluster in a sequential fashion) Factorial study Tests for interactions between interventions; tests multiple factors on an outcome over time Larger number of factors, so greater level of complexity View Large Analysis of the stepped-wedge design can be more complex (eg, interrupted time-series analysis with mixed models that include random and/or fixed effects) and likely requires collaboration with a colleague with statistical expertise [12]. The stepped-wedge design, however, does not directly detect interactions between interventions. To accomplish this feat, one might consider a factorial design. FACTORIAL STUDY DESIGN A factorial study design enables the investigator to analyze the effects and interactions of multiple interventions. In a factorial design, each factor, or intervention, has discrete possible values or “levels.” Factors often have only 2 levels (yes or no); more levels are possible, but they increase the complexity of the design. The execution of a factorial design depends on the number of interventions that one decides to study and the number of levels at each intervention [13]. For simplicity, we use a 2k design (ie, 2 levels for k number of factors) to illustrate the basic principles of factorial designs; however, it is possible to use additional factors to evaluate multiple interventions [14]. A 22 factorial design indicates 2 factors or interventions that each have 2 levels (eg, on or off). For instance, a QI team would like to decrease the long-term use of vancomycin in their hospital unit. They decide to test 2 interventions that can be turned on and off through the electronic health record, (1) an alert that notifies the attending physician when a child has received vancomycin for 72 hours and (2) an alert that notifies the infectious disease pharmacist as soon as vancomycin is ordered for any patient. The QI team would then set up a series of tests (Table 3) and gather data to determine which test performs best in reducing the long-term use of vancomycin in that hospital unit. The data then can be presented as a run or control chart, and the 2 factors also can be presented graphically to determine if an interaction between the factors exists (ie, whether notifying the attending physician and the pharmacist together has a greater effect on the outcome than the sum effect of each notification alone). Thus, the strength of factorial designs includes its ability to test multiple factors on an outcome over time and to test interactions of factors. Table 3. Test No. Factor 1: Alert for Attending Physician Factor 2: Alert for ID Pharmacist 1 On On 2 On Off 3 Off On 4 Off Off Test No. Factor 1: Alert for Attending Physician Factor 2: Alert for ID Pharmacist 1 On On 2 On Off 3 Off On 4 Off Off Abbreviation: ID, infectious diseases. View Large Table 3. Test No. Factor 1: Alert for Attending Physician Factor 2: Alert for ID Pharmacist 1 On On 2 On Off 3 Off On 4 Off Off Test No. Factor 1: Alert for Attending Physician Factor 2: Alert for ID Pharmacist 1 On On 2 On Off 3 Off On 4 Off Off Abbreviation: ID, infectious diseases. View Large CONCLUSIONS Study designs that significantly reduce bias and confounding can strengthen inferences drawn about the effects of interventions used in QI studies. Each study design confers advantages and disadvantages that need to be weighed by the improvement team before starting a QI project. Although simple pre-post test designs are often more feasible for initial improvement efforts, it is important for investigators to be aware that additional study designs exist and not shy away from them when appropriate. Note Potential conflicts of interest. All authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed. References 1. Brady PW , Tchou M , Ambroggio L , et al. Quality improvement feature series article 2: displaying and analyzing quality improvement data . J Pediatric Infect Dis Soc 2017 , in press. 2. Shaughnessy E , Shah A , Ambroggio L , et al. Quality improvement feature series article 1: introduction to quality improvement . J Pediatric Infect Dis Soc 2017 , in press. 3. Harris AD , Lautenbach E , Perencevich E . A systematic review of quasi-experimental study designs in the fields of infection control and antibiotic resistance . Clin Infect Dis 2005 ; 41 : 77 – 82 . Google Scholar Crossref Search ADS PubMed 4. Shonedelmeyer A , Brower L , Statile A , et al. Quality improvement feature series article 3: writing and reviewing quality improvement manuscripts . J Pediatric Infect Dis Soc 2017 , in press. 5. Shadish WR , Cook TD , Campbell DT. Experimental and quasi-experimental designs for generalized causal inference . Boston : Houghton Mifflin ; 2001 . 6. Lopez Bernal J , Cummins S , Gasparrini A . Interrupted time series regression for the evaluation of public health interventions: a tutorial . Int J Epidemiol 2017 ; 46 : 348 – 55 . Google Scholar PubMed 7. Bosso JA , Mauldin PD . Using interrupted time series analysis to assess associations of fluoroquinolone formulary changes with susceptibility of Gram-negative pathogens and isolation rates of methicillin-resistant Staphylococcus aureus . Antimicrob Agents Chemother 2006 ; 50 : 2106 – 12 . Google Scholar Crossref Search ADS PubMed 8. Beilby J , Marley J , Walker D , et al. ; FIESTA Study Group . Effect of changes in antibiotic prescribing on patient outcomes in a community setting: a natural experiment in Australia . Clin Infect Dis 2002 ; 34 : 55 – 64 . Google Scholar Crossref Search ADS PubMed 9. Newman RE , Hedican EB , Herigon JC , et al. Impact of a guideline on management of children hospitalized with community-acquired pneumonia . Pediatrics 2012 ; 129 : e597 – 604 . Google Scholar Crossref Search ADS PubMed 10. Killam WP , Tambatamba BC , Chintu N , et al. Antiretroviral therapy in antenatal care to increase treatment initiation in HIV-infected pregnant women: a stepped-wedge evaluation . AIDS 2010 ; 24 : 85 – 91 . Google Scholar Crossref Search ADS PubMed 11. Brown CA , Lilford RJ . The stepped wedge trial design: a systematic review . BMC Med Res Methodol 2006 ; 6 : 54 . Google Scholar Crossref Search ADS PubMed 12. Twisk JW , Hoogendijk EO , Zwijsen SA , de Boer MR . Different methods to analyze stepped wedge trial designs revealed different aspects of intervention effects . J Clin Epidemiol 2016 ; 72 : 75 – 83 . Google Scholar Crossref Search ADS PubMed 13. Moen RD , Nolan TW , Provost L. Quality Improvement through Planned Experimentation . 3 rd ed. New York : McGraw Hill ; 2012 . 14. Elliott AM , Kizza M , Quigley MA , et al. The impact of helminths on the response to immunization and on the incidence of infection and disease in childhood in Uganda: design of a randomized, double-blind, placebo-controlled, factorial trial of deworming interventions delivered in pregnancy and early childhood . Clin Trials 2007 ; 4 : 42 – 57 . Google Scholar Crossref Search ADS PubMed © The Author(s) 2017. Published by Oxford University Press on behalf of The Journal of the Pediatric Infectious Diseases Society. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)
TI - Quality Improvement Feature Series Article 4: Advanced Designs for Quality Improvement Studies
JF - Journal of the Pediatric Infectious Diseases Society
DO - 10.1093/jpids/pix082
DA - 2018-12-03
UR - https://www.deepdyve.com/lp/oxford-university-press/quality-improvement-feature-series-article-4-advanced-designs-for-VcoApSSjbN
SP - 335
VL - 7
IS - 4
DP - DeepDyve
ER -