Examining the sustainability potential of a multisite pilot to integrate alcohol screening and brief intervention within three primary care systems

Examining the sustainability potential of a multisite pilot to integrate alcohol screening and... Abstract The U.S. Preventive Services Task Force recommends that clinicians adopt universal alcohol screening and brief intervention as a routine preventive service for adults, and efforts are underway to support its widespread dissemination. The likelihood that healthcare systems will sustain this change, once implemented, is under-reported in the literature. This article identifies factors that were important to postimplementation sustainability of an evidence-based practice change to address alcohol misuse that was piloted within three diverse primary care organizations. The Centers for Disease Control and Prevention funded three academic teams to pilot and evaluate implementation of alcohol screening and brief intervention within multiclinic healthcare systems in their respective regions. Following the completion of the pilots, teams used the Program Sustainability Assessment Tool to retrospectively describe and compare differences across eight sustainability domains, identify strengths and potential threats to sustainability, and make recommendations for improvement. Health systems varied across all domains, with greatest differences noted for Program Evaluation, Strategic Planning, and Funding Stability. Lack of funding to sustain practice change, or data monitoring to promote fit and fidelity, was an indication of diminished Organizational Capacity in systems that discontinued the service after the pilot. Early assessment of sustainability factors may identify potential threats that could be addressed prior to, or during implementation to enhance Organizational Capacity. Although this study provides a retrospective assessment conducted by external academic teams, it identifies factors that may be relevant for translating evidence-based behavioral interventions in a way that assures that they are sustained within healthcare systems. Implications Practice: Preimplementation assessment of key sustainability factors will identify potential facilitators and threats, and inform strategies for improving sustained practice change. Policy: National and state healthcare policies could address major threats to sustainability by offering funding or incentives to support implementation start-up activities, and by addressing barriers to billing for screening and brief intervention (SBI) services. Research: Investigation on the impact of assessing and intervening on sustainability domains prior to, or during implementation, is needed. INTRODUCTION Postimplementation sustainability is critical to the success of practice change, yet few organizations identify factors upfront that may threaten practice improvement efforts [1]. Successful initial implementation of a practice change does not assure its long-term sustainability [2], which is defined in the practice change literature as the continuation of a program, with achievement of the same intended outcomes and reach of the target population(s), after initial implementation support has ended [3–6]. Ignoring threats to sustainability wastes resources expended during implementation and undermines translation efforts [7]. Therefore, there is a need for translational science to identify contextual factors that may promote or threaten sustainability, with the goal of improving the likelihood of sustained practice change [8, 9]. Implementation research acknowledges the importance of context in accomplishing practice change [10, 11]; and that each health system, and the clinics within those systems (referred to as sites in this paper), are unique, i.e., what works in one system or site may not work in another [11]. In addition, factors important to sustained change may differ from those associated with successful implementation [12] and may also differ based on the specific practice change being implemented [13]. Thus, comparing diverse healthcare systems engaged in translating a recommended preventive service into their routine practice may help to identify a set of generalizable factors that could be strengthened to improve sustainability and scalability [14] when disseminating the same practice in other settings where conditions may differ [9]. This article retrospectively examines the postimplementation sustainability potential of an evidence-based practice (EBP) that was piloted within three healthcare systems offering primary care services. The implemented change was universal screening and brief intervention (SBI) for alcohol misuse, an EBP with strong evidence of effectiveness, based on over 25 randomized controlled trials demonstrating reductions in binge drinking and improvements in health outcomes [10]. Because of this evidence, the U.S. Preventive Services Task Force (USPSTF) and multiple federal agencies recommend that primary care clinicians adopt alcohol SBI as a preventive service for all adults [15]. Our three pilot projects provided a unique opportunity to: (a) qualitatively explore factor(s) for each setting that contributed to its ability to sustain universal alcohol SBI after implementation support ended; (b) report on the factor(s) that threatened sustainability; and (c) share lessons and strategies that can be used by healthcare systems implementing alcohol SBI to enhance its sustainability. PROJECT CONTEXT AND TIMEFRAME Regional project teams Three academic project teams received Centers for Disease Control and Prevention (CDC) funding in October 2012 to partner with a healthcare system within their respective regions, provide the system with alcohol SBI training and technical assistance, and evaluate the alcohol SBI implementation process. Project teams were located at Meharry Medical College in Nashville, TN, the University of Alaska Anchorage, and the University of Nevada Reno. Project teams ceased providing implementation support to the healthcare systems as of December 2014, with data collection continuing between January and June 2015 to assess program maintenance. In October 2016, to verify if alcohol SBI was sustained 1 year after the grant funding ended, each project team conducted a brief, semistructured follow-up phone interview with each of the health systems. The length of the follow-up interview varied by site but generally lasted fewer than 30 min. Participating health systems Recruitment of the health systems took place in 2012 and 2013. Inclusion criteria were that systems consist of multiple clinics and provide primary care services for adults. The patient population had to include women of reproductive age, in order to promote the utilization of alcohol SBI to prevent alcohol-exposed pregnancies, a key objective of the funder [5]. No financial incentives were provided to the healthcare systems to participate in the pilot project, and no additional staff were hired. Participating systems were: Meharry Medical College Family Medicine System (Fam Med), Alaska Section of Public Health Nursing (Nursing), and University of Nevada School of Medicine Statewide Provider Practice Group (MD-Group). The three health systems differed in organizational structure, services provided, patient demographics, workflow, electronic health record (EHR) capacity and use, and reimbursement. Fam Med was comprised of two family medicine clinics, and operated with centralized decision-making and systems support. Fam Med offered a full-range of primary care services via 16 family physicians and 18 residents in underserved areas of middle Tennessee. Nursing also operated with centralized leadership and policies, to provide prevention (e.g., immunizations) and primary care (e.g., well-child check-ups) services through public health centers located in more than 20 Alaskan communities and through itinerant nurses who travel to 280 towns and villages across Alaska. MD-Group, Nevada’s largest healthcare network, operated as a statewide physician practice group, with 50 independent sites serving more than 155,000 demographically diverse patients in ambulatory clinics and inpatient hospitals. Participating pilot sites Study settings were two Fam Med clinics, three Nursing public health centers (two urban, one rural), and five MD-Group sites that represented family medicine, women’s health, student health, and medical residency clinics. Table 1 summarizes each system and its implementation approach. Table 1 Overview of multi-system implementation pilot for alcohol SBI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI AUDIT Alcohol Use Disorders Identification Test; AUDIT-C the first 3 questions of the AUDIT; CRAFFT Car, Relax, Alone, Forget, Friends, Trouble; EHR electronic health record; MI Motivational Interviewing; RAPS-4 Rapid Alcohol Problems Screen (four question); SBI screening and brief intervention; T-ACE Tolerance, Annoyance, Cut down, Eye opener; TWEAK Tolerance, Worried, Eye opener, Amnesia, and K/Cut down. View Large Table 1 Overview of multi-system implementation pilot for alcohol SBI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI AUDIT Alcohol Use Disorders Identification Test; AUDIT-C the first 3 questions of the AUDIT; CRAFFT Car, Relax, Alone, Forget, Friends, Trouble; EHR electronic health record; MI Motivational Interviewing; RAPS-4 Rapid Alcohol Problems Screen (four question); SBI screening and brief intervention; T-ACE Tolerance, Annoyance, Cut down, Eye opener; TWEAK Tolerance, Worried, Eye opener, Amnesia, and K/Cut down. View Large METHODS Study approach and measures To enable cross-site comparison of factors relevant to sustaining system-level change, each academic project team used the Program Sustainability Assessment Tool (PSAT), a standardized instrument, to rate their system. The PSAT [16, 17] was developed by investigators at Washington University in St. Louis, based on literature review, expert panel input, and concept mapping, with the intention of representing the core domains related to an organization’s capacity to sustain public health programs. Eight domains form the framework, with 40 assessment items divided equally across them. The domains are: Environmental Support (program has strong champions, leadership, public support); Funding Stability (program has stable and sustained funding); Partnerships (community members, organizations, leaders are committed to the program); Organizational Capacity (program is well integrated into operations and supported by leadership, adequate staff, organizational systems); Program Evaluation (program reports short-term and intermediate outcomes, and results inform planning, monitor fidelity to protocols, demonstrate success); Program Adaptation (program periodically reviews the evidence base, adapts to assure continued effectiveness); Communications (communication strategies are in place to maintain public support, market the program, increase awareness of issues it addresses); and Strategic Planning (program plans for future resources and financial needs, and program goals and roles are understood by all stakeholders) [6, 15]. The PSAT domains align well with important influencers of sustainability that were identified in a systematic literature review by Wiltsey Stirman and colleagues [18], as well as factors important to Type 3 (i.e., adapting EBP to real world practice) and Type 4 level translation of prevention interventions (i.e., wide-scale adoption and institutionalization of EBP) [14, 19]. Procedures Program Sustainability Assessment Tool (PSAT) Two raters (Master’s and Doctoral-levels) from each academic project team, who had experienced SBI implementation and involved in most aspects of the implementation pilot, completed a postpilot PSAT assessment, based on observations during meetings and trainings, contact notes from follow-up calls and/or visits, and their own experiences working with their healthcare system partner throughout the project. Raters independently scored each of the 40 PSAT items on a scale of 1 (to little or no extent) to 7 (to a great extent). Domain-specific scores were calculated as the average of the scored items within each domain and an “Overall Capacity for Sustainability” measure was calculated using the average of the eight domain scores for each health system. Individual pilot-site scores were not calculated for Fam Med or Nursing, since all decisions relevant to implementation and sustainability were made centrally. Pilot-site scores were calculated for each of MD-Group’s five sites, since all relevant decisions were decentralized. Regional raters compared item, domain, and overall scores, and discussed discrepancies and observed examples to justify ratings and reach agreement. Scores from each pair’s work were discussed within each regional academic team in order to reach consensus on scores for their health system. Each regional academic team then presented and justified their ratings during a conference call attended by the three teams. In a few instances, recommendations were made to improve cross-site interpretation and consistency in scoring the domains across all rating teams, using the PSAT definitions as a theory-informed framework and taxonomy to guide interpretation [20]. After all sustainability scores were finalized, the three regional academic teams re-convened to compare differences and similarities in scores across the three health systems as well as challenges in scoring the PSAT. Statistical analyses to compare variation in health system PSAT scores were not performed given the small sample and potential for measurement error. Instead, the numerical scores provided a means of qualitatively comparing relative performance on individual domains and to describe sets of domains observed (or not observed) in sustaining sites and nonsustaining sites. Completing and discussing the PSAT also illuminated variation in domain relevance across systems. Although all domains were relevant to Nursing, the MD-Group and Fam Med raters were unable to score the items within the domains of Partnerships and Communications due to their perceived lack of relevance to those systems. As a public health system, Nursing was accountable to numerous community partners and required to garner public support for their services—activities that were uncommon for the other two systems. Thus, these two domains were dropped from the cross-site comparison and the Overall Capacity for Sustainability scores were recalculated based solely on the remaining six domains. (See Table 2 for PSAT scores.) Table 2 Program Sustainability Assessment Tool (PSAT) domain scores by healthcare system Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n aSystem Scores represent the average of the domain item scores for each healthcare system (scale of 1–7), with scores >3.5 (i.e. the midpoint on the PSAT rating scale) interpreted as relatively “strong.” bIndividual Clinic Scores were calculated separately for the five MD-Group clinics, since implementation and sustainability decisions were made at the clinic level. cOverall Capacity for Sustainability scores represent the averages of all average domain scores. dStatus reported by systems during 12-month follow-up call (y = yes; n = no). View Large Table 2 Program Sustainability Assessment Tool (PSAT) domain scores by healthcare system Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n aSystem Scores represent the average of the domain item scores for each healthcare system (scale of 1–7), with scores >3.5 (i.e. the midpoint on the PSAT rating scale) interpreted as relatively “strong.” bIndividual Clinic Scores were calculated separately for the five MD-Group clinics, since implementation and sustainability decisions were made at the clinic level. cOverall Capacity for Sustainability scores represent the averages of all average domain scores. dStatus reported by systems during 12-month follow-up call (y = yes; n = no). View Large Twelve-month postfunding follow-up call After the PSAT data were analyzed and interpreted with regard to sustainability capacity, academic team partners contacted each healthcare system to confirm if alcohol SBI was sustained. One leader at Nursing, two at Fam Med, and one at each of the five MD-Group clinics were asked: (a) if their practice continues to screen all patients for alcohol misuse and conduct brief interventions or make referrals, as needed; (b) whether they were still using the specific screening tool that they were trained on; and (c) what contributed to continuing/discontinuing alcohol SBI at their organization? Responses to these semistructured questions were summarized and answers to question 1 were dichotomized as either yes (y) or no (n) (see Table 2 for sustainability outcomes). Qualitative comparative analysis (QCA) QCA provides a systematic, case-study approach for comparing sets of potentially relevant factors (i.e. conditions), and determining the extent that cases sharing the same conditions also share the same outcome [21]. QCA can be used for small samples and allows for determining potential pathway(s) to success, as well as failure, with regards to achieving the desired outcome [21]. To create each setting’s conditions, PSAT domain scores for Fam Med, Nursing, and MD-Group’s five sites were dichotomized, with scores greater than 3.5 (i.e., the midpoint on the PSAT rating scale) categorized as “strong” and given a value of “1”, and scores equal to or less than 3.5 categorized as “weak” evidence for each domain, and given a value of “0”. Assigning a value of 1 or 0 is interpreted as full inclusion or exclusion in the set (known as “crisp sets”). Although QCA permits the use of “fuzzy sets” (i.e., partial inclusion), the fact that the PSAT domain scale is ordinal, and ratings were based on qualitative data, dichotomizing was justified. Using methods described by Kane and colleagues [21], a table was constructed, with a row for each unique set of conditions (represented by 0 or 1 for each PSAT domain), the number of sites matching those conditions, and the proportion of those sites (from 0.0 to 1.0) that reported sustaining alcohol SBI at 12 months postpilot follow-up (Table 3). This technique helped to illuminate which conditions, by themselves or in combination with others, were likely necessary to sustain alcohol SBI (i.e., the condition was strong in the clinics that sustained and weak in the clinics that did not sustain alcohol SBI); and which condition(s), when present, were likely sufficient to sustain alcohol SBI (i.e., clinics where only a subset of conditions were rated strong, but they still were sustaining alcohol SBI when contacted 12 months following the pilot). Table 3 Qualitative Comparative Analysis of setting conditions and potential relationship to sustainability outcomes. Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 aConditions: A = Fam Med; B = Nursing; MD-Group Clinic 1; C = MD-Group Clinic 2; D = MD-Group Clinics 3–5. bDomain definitions: Env. Supp. (Environmental Support), champions, leadership, public support; Funding Stability, stable and sustained program funding; Org.Cap. (Organizational Capacity), program is well integrated into operations and supported by leadership, adequate staff, systems; Prog. Eval. (Program Evaluation), reports short-term and intermediate outcomes, results inform planning, monitor fidelity to protocols, demonstrate success; Prog. Adapt. (Program Adaptation), program periodically reviews the evidence base, adapts to assure continued effectiveness; and Strategic Planning, plans for future resources and financial needs; program goals and roles are understood by all stakeholders. 1 = strong; 0 = weak evidence for that domain. cSites (n) totaled 7; (Nursing = 1, Fam Med = 1 and MD-Group = 5, denoted as Clinics 1–5). This total reflects the number of sites with the decision-making authority to sustain or not sustain Alcohol SBI. View Large Table 3 Qualitative Comparative Analysis of setting conditions and potential relationship to sustainability outcomes. Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 aConditions: A = Fam Med; B = Nursing; MD-Group Clinic 1; C = MD-Group Clinic 2; D = MD-Group Clinics 3–5. bDomain definitions: Env. Supp. (Environmental Support), champions, leadership, public support; Funding Stability, stable and sustained program funding; Org.Cap. (Organizational Capacity), program is well integrated into operations and supported by leadership, adequate staff, systems; Prog. Eval. (Program Evaluation), reports short-term and intermediate outcomes, results inform planning, monitor fidelity to protocols, demonstrate success; Prog. Adapt. (Program Adaptation), program periodically reviews the evidence base, adapts to assure continued effectiveness; and Strategic Planning, plans for future resources and financial needs; program goals and roles are understood by all stakeholders. 1 = strong; 0 = weak evidence for that domain. cSites (n) totaled 7; (Nursing = 1, Fam Med = 1 and MD-Group = 5, denoted as Clinics 1–5). This total reflects the number of sites with the decision-making authority to sustain or not sustain Alcohol SBI. View Large RESULTS Conditions supporting sustainability Twelve-month postfunding follow-up calls confirmed sustainment of alcohol SBI as implemented for Fam Med, Nursing, and one MD-Group site. Sustaining sites were scored as strong on at least five of the six PSAT domains: Environmental Support, Program Adaptation, Organizational Capacity, Program Evaluation, and Strategic Planning (Table 2). The consistency in strengths across the two Fam Med and three Nursing sites was likely due to the involvement of centralized leaders and internal champions in the planning, piloting, and sustainability activities. The expertise of Fam Med’s lead project personnel, who were both providers within the Fam Med system and researchers in the field of alcohol misuse, fostered routine review of the science, which was then used in conjunction with findings from their internal quality assurance processes to update trainings and make adaptations that enhanced the overall feasibility and effectiveness of procedures. Fam Med incorporated alcohol SBI into its standard operating procedures through its training, quality improvement activities, and electronic health record (EHR) integration. Despite this success in integrating alcohol SBI into its practice workflows and EHR, Fam Med relayed some uncertainty regarding continuing training, performance monitoring, and billing at the end of the pilot. Also, a change in Fam Med’s leadership affected program support during the transitional period, highlighting a potential threat to sustaining alcohol SBI and indicating a need to engage in more strategic planning. Nursing planning team members drafted a written policy and procedure during the initial planning process, and presented it to Nursing leadership for approval. The intention was to refine the draft, throughout the implementation pilot, as adaptations were made. Nurses at the pilot sites were informed that their input would help shape the final policy and procedure prior to its dissemination, thereby improving its integration into operations, and ultimate acceptance as standardized practice. Drafting, piloting, and refining the written policy is evidence of Nursing’s willingness to make adaptations that improved its Organizational Capacity for sustaining alcohol SBI. Taking the time to assure the policy and protocols were feasible and acceptable to providers likely contributed to Nursing routinizing alcohol SBI, despite provider turnover, changes in nurse managers, and retirement of the administrative leader during the pilot period. Towards the end of the pilot, Nursing disseminated alcohol SBI statewide, across all clinics and itinerant services. Alternatively, MD-Group’s decentralized structure likely contributed to the variation observed across the five pilot sites, since operational decisions were made at the practice level, thereby limiting MD-Group’s capacity to provide the support necessary to sustain alcohol SBI, such as continued staff training and performance monitoring. Two sites (Clinic 1 and Clinic 2) were scored strong on four domains, but only Clinic 1, which added standardized screening questions as required fields within its EHR’s health history form, sustained its alcohol SBI protocols. Funding Stability may also have been a contributing factor, given that Clinic 1, unlike the other MD-Group sites, was not dependent on billing or grants to sustain its activities. Use of process evaluation data informed meaningful program adaptation Our qualitative comparative analysis (Table 3) identified only one factor that was consistently strong for all sustaining sites and consistently weak for all nonsustaining sites—Program Evaluation. During the implementation pilot, all regional teams requested monthly data to track screening rates at the system and pilot site levels, and monitor, to the extent possible, fidelity to alcohol SBI protocols. For the nonsustaining sites, data extraction challenges were pervasive and limited MD-Group’s ability to monitor screening and follow-up services with sufficient regularity to identify issues or areas in need of improvement. For example, three of the sites did not document screening directly in the health records, thereby relying on paper forms and manual data entry. Clinic managers reported that it was too time intensive to track and report these data. For the sustaining sites, modifications were made to the patient database (Nursing) or EHR (Fam Med and MD-Group’s Clinic 1), providing a systematic way to document patient screening results and extract data. For these sites, the use of data for reporting progress on health indicators was valued. For example, as a student health center, Clinic 1, which participated in the American College Health Association-National College Health Assessment (ACHA-NCHA), expressed interest in comparing their alcohol screening results to ACHA-NCHA outcomes. Despite indications that sites rated as strong in Program Evaluation were successful at sustaining alcohol SBI and sites rated as weak were unsuccessful at sustaining, we cannot assume that engaging in data collection and reporting outcomes is sufficient for sustaining change. We observed that the systems using fidelity indicators that they monitored over time could iteratively identify issues and make meaningful adaptations that increased their organizational capacity to sustain change. For example, Fam Med and MD-Group’s Clinic 1 used their EHR to track screening rates and monitor compliance with scoring intervention, and documentation protocols, that were then addressed by system leadership. Although it lacked an EHR, Nursing’s database was modified to include entry of screenings completed or deferred and positive scores, from which a monthly screening report was produced. The report indicated a steep decline in screening rates for one of its pilot sites about 4 months after implementation, indicating a lack of adherence to the draft policy and procedures. In addition, feedback collected by academic project staff during scheduled semistructured follow-up calls revealed that nurses at all pilot sites lacked confidence in conducting brief interventions. Skills-focused booster training was provided at all sites to address both issues. Conversely, four MD-Group sites lacked an electronic mechanism for monitoring and reporting data. Without a systematic way to hold providers and staff accountable to established protocols, data-driven adaptations to improve feasibility or fidelity were lacking. Hence, the inability or unwillingness to engage in program evaluation, or to use the data that are generated to refine processes during implementation, may signal a serious threat to sustainability. Organizational capacity to sustain change not necessarily limited by funding instability Weak funding stability was a common factor at all nonsustaining sites. However, funding was also a concern for one sustaining site, suggesting the compensatory strength of other factors in the face of financial instability. For example, challenges related to funding stability may potentially be overcome by capacity to garner commitment, and by adapting programs to assure that they are efficiently integrated into existing workflows and systems. MD-Group’s lower Funding Stability ratings reflected the inability of three of its five pilot sites to be reimbursed due to alcohol SBI Medicaid billing codes not being activated in Nevada. Another of the sites (Clinic 2), which is funded primarily through grants, also did not sustain alcohol SBI, despite being rated as strong on four of the six domains. MD-Group’s nonsustaining clinics implemented very few adaptations to improve alcohol SBI’s feasibility, and suggestions for process improvement were not successfully integrated into workflows. For example, a recommendation by a clinic manager to create a note template that would enhance provider follow-up on interventions and referrals was not successful since providers were not required to use it. Thus, although the nonsustaining sites acknowledged alcohol SBI’s importance, they did not identify any strategies that would enable them to sustain it without third party reimbursement. On the other hand, Fam Med, a sustaining system, also reported long-term funding challenges, due to its reliance on federal funding to support implementation and personnel costs associated with the training and oversight of the implementation pilot. Although Fam Med embedded the appropriate billing codes in the EHR, its clinical staff and project personnel identified barriers related to reimbursement during the pilot. One barrier was the inability to code for alcohol SBI and a problem-oriented visit on the same day. Despite this serious threat to sustainability, Fam Med continues to provide alcohol SBI, largely due to its success in the other domains, which expanded their organizational capacity. Thus, in the presence of other strengths, a lack of Funding Stability does not destine a setting for failure. Lessons from health systems on sustainability Twelve-month postfunding follow-up interviews with leaders at each setting confirmed their success in sustaining alcohol SBI a full year after the pilot projects ended, and provided an opportunity to further understand the PSAT results. Their answers are summarized below. Fam Med reported that both clinics in their health system continue to screen all patients ages 18 and older for alcohol misuse every 6 months and provide a brief intervention or referral to treatment, as indicated by the screening results. According to the Fam Med healthcare providers, they have continued alcohol SBI at their system due, in large part, to the belief that this type of evidence-based prevention strategy should be standard practice throughout primary care and in related healthcare encounters. One provider said, “Incorporating the process into the EHR, including using health reminders and prompts, was key to standardizing this practice in our clinics.” They note that in an underserved community, such as theirs, it is important to educate patients about risky health behaviors and assist them with seeking outside treatment. Nursing’s section chief credited a number of factors to the overall sustainment of their alcohol SBI program, many of which confirmed PSAT identified strengths. First, she reported that engaging staff in planning, and piloting alcohol SBI prior to system-wide dissemination, allowed them to identify glitches, improve their procedures, and create tools and reports that helped support the implementation process. Second, ongoing standardization and consistency among all staff was achieved by adapting alcohol SBI training modules that could be accessed through their existing electronic learning platform, completion of which is required for all new providers. Last, she described how applying lessons learned from implementing other screenings helped make this one work, saying, “We knew what worked for our DV [Domestic Violence] screening and how that was implemented.” Of the five pilot clinics within MD-Group only Clinic 1, the setting that integrated the screening tool into their EHR system and that was funded through student fees (i.e., not dependent on grants or third party reimbursement), reported sustaining a universal approach to screening and intervening for alcohol misuse, using the screening questions and tools adopted during the implementation pilot. They reported that having the screening tool in the EHR was vital to facilitating the process. The other sites reported that they screen for alcohol sporadically and cited “time factors” as the main barrier to sustaining universal alcohol SBI. DISCUSSION Organizational capacity to sustain change relies on multiple factors that, taken together, contribute to integrating evidence-based interventions into practice. We found that the sites that used their process data to inform adaptations that enhanced the feasibility and fidelity of alcohol SBI, also enhanced their capacity to keep it going after the pilot study was completed. Research describing the barriers and challenges for implementing practice change is plentiful; however, less attention has been given to specifying factors that influence whether changes will be sustained after implementation. Facilitators to practice change, such as provider knowledge and skills training, may themselves be unsustainable if the desired provider behavior change is not supported by organizational infrastructure or protocols [12, 17, 22, 23]. This paper used a sustainability framework and assessment tool and QCA to analyze and compare three regional healthcare systems on six domains that have been associated with sustaining public health programs. The tool provided a common empirical framework and taxonomy from which we used qualitative methods to compare factors that may be necessary and those that may be sufficient to achieving sustainability. While implementation approaches across the sites were similar, a few differences in setting conditions were observed that may have contributed to sustainability success. First, all of the nonsustaining systems were rated low on Funding Stability, a barrier to implementation and sustained change that is frequently reported in the alcohol SBI literature [24]. However, Funding Stability was also a concern in one of the sustaining systems, suggesting that their efforts during implementation to fully imbed alcohol SBI into their practice priorities and workflows enhanced their Organizational Capacity to sustain the change [25, 26]. Second, the practice change literature emphasizes the importance of using data as a process improvement strategy to manage change [27], yet the importance of ongoing program evaluation is seldom identified as a barrier to program sustainability. In our study, the domain of Program Evaluation was a key difference between the sites that sustained alcohol SBI and those that did not. Although several sites used electronic databases or EHRs to cue providers to screen patients, the extent that system-level data were used to monitor, report on, and make program adaptations that improved process fidelity varied. It is interesting to note that Nursing and Fam Med used a paper screening form that involved later data entry into the patient data base or EHR. This did not appear to threaten sustainability, however, given this modality matched their other health risk screenings. What was most important was having a documentation and data extraction protocol to monitor process fidelity. Identifying even one metric that could be monitored monthly, such as percent of eligible patients or visits screened, along with soliciting staff feedback and recommendations to refine protocols, provides a means of identifying problems and improving the logistical fit and sustainment of the intervention within a practice [28–30]. Third, a centralized leadership structure, where decision-making authority for the whole system was concentrated at the top, was associated with sustainability among sites in this study, and decentralized leadership, where decision-making authority to make changes to business operations was held by individual clinic managers, was identified as a barrier to sustainability. Although a study of the influence of organizational dynamics on capacity for implementing sustained practice change is beyond the scope of this paper [31], the ability to achieve standardization and consistency of implementation within a centralized structure may or may not outweigh the importance of achieving the local-level buy-in required by a decentralized structure [32]. It is noteworthy that Nursing and Fam Med’s centralized leadership involved clinic managers and staff in the planning and piloting process of the implementation, promoting both standardization and buy-in. Such collaborative decision-making practices, which foster participation and provide opportunities to provide input, have been strongly associated with sustainability [25]. Decentralized systems may likewise expand their capacity for practice improvement by collaborating with other providers to combine resources and strategies [33]. Thus, proactively compensating for the limitations that may be inherent within centralized or decentralized systems could improve both standardization and acceptance of the practice change. Finally, our follow-up calls with the health system leaders, a year after the project was completed, confirmed that several key PSAT factors were also identified by the health systems as contributing to sustaining alcohol SBI. In addition, health systems mentioned three important factors that may not have been readily observed by external raters: (a) utilizing internal knowledge of what has worked (or not worked) historically with regard to implementing new services may enhance program implementation and proactively address issues that will improve the likelihood of sustainability [34]; (b) acknowledging that alcohol SBI addresses a gap in service provision may improve staff acceptance; and (c) a fundamental belief that providing universal alcohol SBI is aligned with the practice’s commitment to provide appropriate, evidence-based patient care may override deficits in other domains. Although the use of a standardized framework strengthened our ability to focus on domains identified in the literature as important to sustainability, two of the PSAT domains were not considered relevant to the nonpublic health primary care sites, a limitation of using an instrument designed for public health. In addition, the full use of QCA to compare the extent that diverse sites with similar conditions share the same outcome was limited by our sample size and number of domains. With only seven cases that potentially varied on six different dimensions, not all possible combinations of factors were observed. So, although our findings did identify one domain that may be necessary for sustainability (i.e., Program Evaluation), more research on the potential pathways to sustained change is required. Also, our decision to dichotomize the domains may have obscured the potential influence of domains that were somewhat or partially present, as well as their different levels of importance within each system. Our study provided a retrospective assessment conducted by program facilitators as opposed to the healthcare systems themselves. Although raters were involved in most aspects of implementation and had documented observations and feedback provided by the health system and clinic staff throughout the study, comparing their ratings with healthcare system staff would strengthen our findings. On the other hand, our 12-month, post-pilot follow-up calls to system leaders confirmed sustainability outcomes and provided a valuable organizational perspective. CONCLUSIONS AND RECOMMENDATIONS The ability of health systems and implementation researchers to assess and address threats to sustaining practice change as part of implementation planning will strengthen the likelihood that the reach and quality of newly adopted practices are sustained. Consistent with the notion that all healthcare settings are unique, the pathways for successfully sustaining practice change do not appear to conform to one set of conditions. Understanding the domains that contribute to success or failure, and identifying ways to remove the threats or mitigate their influence by strengthening other areas, broadens the potential for success. We recommend that future research explores prospective assessment of the relevant sustainability domains by healthcare systems to determine whether directly addressing potential threats to sustainability (e.g., locating funding sources, shoring up program evaluation capabilities/requirements), or strengthening other domains to compensate, (e.g., adapting programs to assure they are well integrated into workflows and practice standards) results in sustained change. Research that replicates and expands on our results, using a larger sample, would further define the conditions that are necessary and/or sufficient for successfully sustaining practice change. In addition, mediator/moderator analyses to better understand the pathways leading to sustainability would enhance our ability to assess and intervene more precisely. Finally, we recommend that funders and policymakers consider the importance of providing start-up funding, technical assistance, and practical quality indicators that can be used to sustain quality processes and outcomes [19]. Acknowledgments This publication was supported by funding from the U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, cooperative agreement numbers U84DD000882, U84DD000886, and U84DD000888, funding opportunity DD11-1107. The findings and conclusions in this publication are those of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention. Compliance with Ethical Standards Conflict of interest: D.K. King, S.J. Gonzalez, J. Hartje, B.L. Hanson, C. Edney, H. Snell, R.J. Zoorob, and N.A. Roget have no conflict of interest to report. References 1. Minnier TE . How To Build Sustainability Into the Innovation Process ; 2014 . Available at https://innovations.ahrq.gov/perspectives/how-build-sustainability-innovation-process. Accessibility verified March 24, 2014 . 2. Fleiszer AR , Semenic SE , Ritchie JA , Richer MC , Denis JL . The sustainability of healthcare innovations: a concept analysis . J Adv Nurs . 2015 ; 71 ( 7 ): 1484 – 1498 . Google Scholar Crossref Search ADS PubMed 3. Pagoto SL , Spring B , Coups EJ , Mulvaney S , Coutu MF , Ozakinci G . Barriers and facilitators of evidence-based practice perceived by behavioral science health professionals . J Clin Psychol . 2007 ; 63 ( 7 ): 695 – 705 . Google Scholar Crossref Search ADS PubMed 4. Tabak RG , Khoong EC , Chambers DA , Brownson RC . Bridging research and practice: models for dissemination and implementation research . Am J Prev Med . 2012 ; 43 ( 3 ): 337 – 350 . Google Scholar Crossref Search ADS PubMed 5. Johnson K , Hays C , Center H , Daley C . Building capacity and sustainable prevention innovations: a sustainability planning model . Eval Program Plann . 2004 ; 27 ( 2 ): 135 – 149 . Google Scholar Crossref Search ADS 6. Brown PM , Cameron LD , Ramondt S . Sustainability of behavioral interventions: beyond cost-effectiveness analysis . Int J Behav Med . 2015 ; 22 ( 3 ): 425 – 433 . Google Scholar Crossref Search ADS PubMed 7. Fleiszer AR , Semenic SE , Ritchie JA , Richer MC , Denis JL . An organizational perspective on the long-term sustainability of a nursing best practice guidelines program: a case study . BMC Health Serv Res . 2015 ; 15 : 535 . Google Scholar Crossref Search ADS PubMed 8. Shediac-Rizkallah MC , Bone LR . Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy . Health Educ Res . 1998 ; 13 ( 1 ): 87 – 108 . Google Scholar Crossref Search ADS PubMed 9. Scheirer MA , Dearing JW . An agenda for research on the sustainability of public health programs . Am J Public Health . 2011 ; 101 ( 11 ): 2059 – 2067 . Google Scholar Crossref Search ADS PubMed 10. Kaner EF , Dickinson HO , Beyer F , et al. The effectiveness of brief alcohol interventions in primary care settings: a systematic review . Drug Alcohol Rev . 2009 ; 28 ( 3 ): 301 – 323 . Google Scholar Crossref Search ADS PubMed 11. Armstrong N , Herbert G , Brewster L . Contextual barriers to implementation in primary care: an ethnographic study of a programme to improve chronic kidney disease care . Fam Pract . 2016 ; 33 ( 4 ): 426 – 431 . Google Scholar Crossref Search ADS PubMed 12. Hunter SB , Han B , Slaughter ME , Godley SH , Garner BR . Associations between implementation characteristics and evidence-based practice sustainment: a study of the Adolescent Community Reinforcement Approach . Implement Sci . 2015 ; 10 : 173 . Google Scholar Crossref Search ADS PubMed 13. Scheirer MA . Linking sustainability research to intervention types . Am J Public Health . 2013 ; 103 ( 4 ): e73 – e80 . Google Scholar Crossref Search ADS PubMed 14. Hassmiller Lich K , Frerichs L , Fishbein D , Bobashev G , Pentz MA . Translating research into prevention of high-risk behaviors in the presence of complex systems: definitions and systems frameworks . Transl Behav Med . 2016 ; 6 ( 1 ): 17 – 31 . Google Scholar Crossref Search ADS PubMed 15. Moyer VA ; Preventive Services Task Force . Screening and behavioral counseling interventions in primary care to reduce alcohol misuse: U.S. preventive services task force recommendation statement . Ann Intern Med . 2013 ; 159 ( 3 ): 210 – 218 . Google Scholar PubMed 16. Luke DA , Calhoun A , Robichaux CB , Elliott MB , Moreland-Russell S . The Program Sustainability Assessment Tool: a new instrument for public health programs . Prev Chronic Dis . 2014 ; 11 : 130184 . Google Scholar Crossref Search ADS PubMed 17. Calhoun A , Mainor A , Moreland-Russell S , Maier RC , Brossart L , Luke DA . Using the Program Sustainability Assessment Tool to assess and plan for sustainability . Prev Chronic Dis . 2014 ; 11 : 130185 . Google Scholar Crossref Search ADS PubMed 18. Wiltsey Stirman S , Kimberly J , Cook N , Calloway A , Castro F , Charns M . The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research . Implement Sci . 2012 ; 7 : 17 . Google Scholar Crossref Search ADS PubMed 19. Aldridge WA II , Boothroyd RI , Fleming WO , et al. Transforming community prevention systems for sustained impact: embedding active implementation and scaling functions . Transl Behav Med . 2016 ; 6 ( 1 ): 135 – 144 . Google Scholar Crossref Search ADS PubMed 20. Bradley EH , Curry LA , Devers KJ . Qualitative data analysis for health services research: developing taxonomy, themes, and theory . Health Serv Res . 2007 ; 42 ( 4 ): 1758 – 1772 . Google Scholar Crossref Search ADS PubMed 21. Kane H , Lewis MA , Williams PA , Kahwati LC . Using qualitative comparative analysis to understand and quantify translation and implementation . Transl Behav Med . 2014 ; 4 ( 2 ): 201 – 208 . Google Scholar Crossref Search ADS PubMed 22. Virani T , Lemieux-Charles L , Davis DA , Berta W . Sustaining change: once evidence-based practices are transferred, what then ? Healthc Q . 2009 ; 12 ( 1 ): 89 – 96, 2 . Google Scholar Crossref Search ADS PubMed 23. Seale JP , Velasquez MM , Johnson JA , et al. Skills-based residency training in alcohol screening and brief intervention: results from the Georgia-Texas “Improving Brief Intervention” Project . Subst Abus . 2012 ; 33 ( 3 ): 261 – 271 . Google Scholar Crossref Search ADS PubMed 24. Bernstein E , Topp D , Shaw E , et al. A preliminary report of knowledge translation: lessons from taking screening and brief intervention techniques from the research setting into regional systems of care . Acad Emerg Med . 2009 ; 16 ( 11 ): 1225 – 1233 . Google Scholar Crossref Search ADS PubMed 25. Durlak JA , DuPre EP . Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation . Am J Community Psychol . 2008 ; 41 ( 3–4 ): 327 – 350 . Google Scholar Crossref Search ADS PubMed 26. Fixsen DL , Naoom SF , Blase K , Friedman RM , Wallace F. Implementation research: a synthesis of the literature . Tampa, FL : Louis de la Parte FL Mental Health Inst .; 2005 . 27. Clark KD , Miller BF , Green LA , de Gruy FV , Davis M , Cohen DJ . Implementation of behavioral health interventions in real world scenarios: Managing complex change . Fam Syst Health . 2017 ; 35 ( 1 ): 36 – 45 . Google Scholar Crossref Search ADS PubMed 28. Cooper BR , Bumbarger BK , Moore JE . Sustaining evidence-based prevention programs: correlates in a large-scale dissemination initiative . Prev Sci . 2015 ; 16 ( 1 ): 145 – 157 . Google Scholar Crossref Search ADS PubMed 29. Moore JE , Bumbarger BK , Cooper BR . Examining adaptations of evidence-based programs in natural contexts . J Prim Prev . 2013 ; 34 ( 3 ): 147 – 161 . Google Scholar Crossref Search ADS PubMed 30. Elliott DS , Mihalic S . Issues in disseminating and replicating effective prevention programs . Prev Sci . 2004 ; 5 ( 1 ): 47 – 53 . Google Scholar Crossref Search ADS PubMed 31. Harlos K , Tetroe J , Graham ID , Bird M , Robinson N . Mining the management literature for insights into implementing evidence-based change in healthcare . Healthc Policy . 2012 ; 8 ( 1 ): 33 – 48 . Google Scholar PubMed 32. Davis MV , Mahanna E , Joly B , et al. Creating quality improvement culture in public health agencies . Am J Public Health . 2014 ; 104 ( 1 ): e98 – 104 . Google Scholar Crossref Search ADS PubMed 33. Lanter PL , Wolff KB , Johnson LC , Ercolano EM , Kilmer JR , Provost L . Change IS Possible: Reducing High-Risk Drinking Using a Collaborative Improvement Model . J Am Coll Health . 2015 ; 63 ( 5 ): 330 – 336 . Google Scholar Crossref Search ADS PubMed 34. Sackmann SA , Eggenhofer-Rehart PM , Friesl M . Sustainable Change: Long-term efforts toward developing a learning organization . J Appl Behav Sci . 2009 ; 45 ( 4 ): 521 – 549 . Google Scholar Crossref Search ADS © Society of Behavioral Medicine 2018. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Translational Behavioral Medicine Oxford University Press

Examining the sustainability potential of a multisite pilot to integrate alcohol screening and brief intervention within three primary care systems

Loading next page...
 
/lp/ou_press/examining-the-sustainability-potential-of-a-multisite-pilot-to-rcM90w0R6A
Copyright
© Society of Behavioral Medicine 2018. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com
ISSN
1869-6716
eISSN
1613-9860
D.O.I.
10.1093/tbm/ibx020
Publisher site
See Article on Publisher Site

Abstract

Abstract The U.S. Preventive Services Task Force recommends that clinicians adopt universal alcohol screening and brief intervention as a routine preventive service for adults, and efforts are underway to support its widespread dissemination. The likelihood that healthcare systems will sustain this change, once implemented, is under-reported in the literature. This article identifies factors that were important to postimplementation sustainability of an evidence-based practice change to address alcohol misuse that was piloted within three diverse primary care organizations. The Centers for Disease Control and Prevention funded three academic teams to pilot and evaluate implementation of alcohol screening and brief intervention within multiclinic healthcare systems in their respective regions. Following the completion of the pilots, teams used the Program Sustainability Assessment Tool to retrospectively describe and compare differences across eight sustainability domains, identify strengths and potential threats to sustainability, and make recommendations for improvement. Health systems varied across all domains, with greatest differences noted for Program Evaluation, Strategic Planning, and Funding Stability. Lack of funding to sustain practice change, or data monitoring to promote fit and fidelity, was an indication of diminished Organizational Capacity in systems that discontinued the service after the pilot. Early assessment of sustainability factors may identify potential threats that could be addressed prior to, or during implementation to enhance Organizational Capacity. Although this study provides a retrospective assessment conducted by external academic teams, it identifies factors that may be relevant for translating evidence-based behavioral interventions in a way that assures that they are sustained within healthcare systems. Implications Practice: Preimplementation assessment of key sustainability factors will identify potential facilitators and threats, and inform strategies for improving sustained practice change. Policy: National and state healthcare policies could address major threats to sustainability by offering funding or incentives to support implementation start-up activities, and by addressing barriers to billing for screening and brief intervention (SBI) services. Research: Investigation on the impact of assessing and intervening on sustainability domains prior to, or during implementation, is needed. INTRODUCTION Postimplementation sustainability is critical to the success of practice change, yet few organizations identify factors upfront that may threaten practice improvement efforts [1]. Successful initial implementation of a practice change does not assure its long-term sustainability [2], which is defined in the practice change literature as the continuation of a program, with achievement of the same intended outcomes and reach of the target population(s), after initial implementation support has ended [3–6]. Ignoring threats to sustainability wastes resources expended during implementation and undermines translation efforts [7]. Therefore, there is a need for translational science to identify contextual factors that may promote or threaten sustainability, with the goal of improving the likelihood of sustained practice change [8, 9]. Implementation research acknowledges the importance of context in accomplishing practice change [10, 11]; and that each health system, and the clinics within those systems (referred to as sites in this paper), are unique, i.e., what works in one system or site may not work in another [11]. In addition, factors important to sustained change may differ from those associated with successful implementation [12] and may also differ based on the specific practice change being implemented [13]. Thus, comparing diverse healthcare systems engaged in translating a recommended preventive service into their routine practice may help to identify a set of generalizable factors that could be strengthened to improve sustainability and scalability [14] when disseminating the same practice in other settings where conditions may differ [9]. This article retrospectively examines the postimplementation sustainability potential of an evidence-based practice (EBP) that was piloted within three healthcare systems offering primary care services. The implemented change was universal screening and brief intervention (SBI) for alcohol misuse, an EBP with strong evidence of effectiveness, based on over 25 randomized controlled trials demonstrating reductions in binge drinking and improvements in health outcomes [10]. Because of this evidence, the U.S. Preventive Services Task Force (USPSTF) and multiple federal agencies recommend that primary care clinicians adopt alcohol SBI as a preventive service for all adults [15]. Our three pilot projects provided a unique opportunity to: (a) qualitatively explore factor(s) for each setting that contributed to its ability to sustain universal alcohol SBI after implementation support ended; (b) report on the factor(s) that threatened sustainability; and (c) share lessons and strategies that can be used by healthcare systems implementing alcohol SBI to enhance its sustainability. PROJECT CONTEXT AND TIMEFRAME Regional project teams Three academic project teams received Centers for Disease Control and Prevention (CDC) funding in October 2012 to partner with a healthcare system within their respective regions, provide the system with alcohol SBI training and technical assistance, and evaluate the alcohol SBI implementation process. Project teams were located at Meharry Medical College in Nashville, TN, the University of Alaska Anchorage, and the University of Nevada Reno. Project teams ceased providing implementation support to the healthcare systems as of December 2014, with data collection continuing between January and June 2015 to assess program maintenance. In October 2016, to verify if alcohol SBI was sustained 1 year after the grant funding ended, each project team conducted a brief, semistructured follow-up phone interview with each of the health systems. The length of the follow-up interview varied by site but generally lasted fewer than 30 min. Participating health systems Recruitment of the health systems took place in 2012 and 2013. Inclusion criteria were that systems consist of multiple clinics and provide primary care services for adults. The patient population had to include women of reproductive age, in order to promote the utilization of alcohol SBI to prevent alcohol-exposed pregnancies, a key objective of the funder [5]. No financial incentives were provided to the healthcare systems to participate in the pilot project, and no additional staff were hired. Participating systems were: Meharry Medical College Family Medicine System (Fam Med), Alaska Section of Public Health Nursing (Nursing), and University of Nevada School of Medicine Statewide Provider Practice Group (MD-Group). The three health systems differed in organizational structure, services provided, patient demographics, workflow, electronic health record (EHR) capacity and use, and reimbursement. Fam Med was comprised of two family medicine clinics, and operated with centralized decision-making and systems support. Fam Med offered a full-range of primary care services via 16 family physicians and 18 residents in underserved areas of middle Tennessee. Nursing also operated with centralized leadership and policies, to provide prevention (e.g., immunizations) and primary care (e.g., well-child check-ups) services through public health centers located in more than 20 Alaskan communities and through itinerant nurses who travel to 280 towns and villages across Alaska. MD-Group, Nevada’s largest healthcare network, operated as a statewide physician practice group, with 50 independent sites serving more than 155,000 demographically diverse patients in ambulatory clinics and inpatient hospitals. Participating pilot sites Study settings were two Fam Med clinics, three Nursing public health centers (two urban, one rural), and five MD-Group sites that represented family medicine, women’s health, student health, and medical residency clinics. Table 1 summarizes each system and its implementation approach. Table 1 Overview of multi-system implementation pilot for alcohol SBI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI AUDIT Alcohol Use Disorders Identification Test; AUDIT-C the first 3 questions of the AUDIT; CRAFFT Car, Relax, Alone, Forget, Friends, Trouble; EHR electronic health record; MI Motivational Interviewing; RAPS-4 Rapid Alcohol Problems Screen (four question); SBI screening and brief intervention; T-ACE Tolerance, Annoyance, Cut down, Eye opener; TWEAK Tolerance, Worried, Eye opener, Amnesia, and K/Cut down. View Large Table 1 Overview of multi-system implementation pilot for alcohol SBI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI Health systems Fam Med Nursing MD-Group Pilot sites (n) 2 3 5 Planning process -One centralized SBI planning team included clinic directors, nurse manager, social worker, alcohol SBI coordinator. -The team designed protocols, solicited input from receptionists, medical assistant, and nursing. -One centralized SBI planning team included regional directors, nurse managers, quality assurance, data manager, project team. -The team selected screening tools; agreed on frequency, visit type, workflow; and drafted a policy and procedure. -Five distinct SBI planning teams were created. Each team was composed of project and clinic staff. -The teams each selected screening tools; agreed on frequency and criteria for patient screening. Implementation outcome evaluation Reach: monthly screening rates, obtained (EHR) Reach and fidelity to protocols: monthly screening rates and chart review (database) Reach: quarterly screening data (Site manager completed forms created by project team) Implementation process evaluation -Forms and processes pre-tested with a few patients; revised per staff feedback -Semistructured facilitation calls with each site; data were used to iteratively refine policy -Phone follow-up and annual site visits; processes revised per feedback Population screened Male and female adults (ages 18 and older) Male and female adults (ages 18 and older); adolescents (ages 13–17) Male and female adults (ages 18 and older) Screenera AUDIT; AUDIT-C T-ACE (adults); CRAFFT (adolescents); two quantity/frequency items AUDIT-C; TWEAK; RAPS-4 Visits receiving SBI Primary care Gynecology, family planning, prenatal, post-partum, STI/STD, tuberculosis General medical/acute conditions, wellness, well-woman, gynecological, prenatal Screening frequency Annually, then every 6 months Every visit (monthly for tuberculosis visits) Annually Brief intervention Intervention based on AUDIT zones I–IV (low risk to dependent) Reinforcement and education using MI principles Reinforcement and education using MI AUDIT Alcohol Use Disorders Identification Test; AUDIT-C the first 3 questions of the AUDIT; CRAFFT Car, Relax, Alone, Forget, Friends, Trouble; EHR electronic health record; MI Motivational Interviewing; RAPS-4 Rapid Alcohol Problems Screen (four question); SBI screening and brief intervention; T-ACE Tolerance, Annoyance, Cut down, Eye opener; TWEAK Tolerance, Worried, Eye opener, Amnesia, and K/Cut down. View Large METHODS Study approach and measures To enable cross-site comparison of factors relevant to sustaining system-level change, each academic project team used the Program Sustainability Assessment Tool (PSAT), a standardized instrument, to rate their system. The PSAT [16, 17] was developed by investigators at Washington University in St. Louis, based on literature review, expert panel input, and concept mapping, with the intention of representing the core domains related to an organization’s capacity to sustain public health programs. Eight domains form the framework, with 40 assessment items divided equally across them. The domains are: Environmental Support (program has strong champions, leadership, public support); Funding Stability (program has stable and sustained funding); Partnerships (community members, organizations, leaders are committed to the program); Organizational Capacity (program is well integrated into operations and supported by leadership, adequate staff, organizational systems); Program Evaluation (program reports short-term and intermediate outcomes, and results inform planning, monitor fidelity to protocols, demonstrate success); Program Adaptation (program periodically reviews the evidence base, adapts to assure continued effectiveness); Communications (communication strategies are in place to maintain public support, market the program, increase awareness of issues it addresses); and Strategic Planning (program plans for future resources and financial needs, and program goals and roles are understood by all stakeholders) [6, 15]. The PSAT domains align well with important influencers of sustainability that were identified in a systematic literature review by Wiltsey Stirman and colleagues [18], as well as factors important to Type 3 (i.e., adapting EBP to real world practice) and Type 4 level translation of prevention interventions (i.e., wide-scale adoption and institutionalization of EBP) [14, 19]. Procedures Program Sustainability Assessment Tool (PSAT) Two raters (Master’s and Doctoral-levels) from each academic project team, who had experienced SBI implementation and involved in most aspects of the implementation pilot, completed a postpilot PSAT assessment, based on observations during meetings and trainings, contact notes from follow-up calls and/or visits, and their own experiences working with their healthcare system partner throughout the project. Raters independently scored each of the 40 PSAT items on a scale of 1 (to little or no extent) to 7 (to a great extent). Domain-specific scores were calculated as the average of the scored items within each domain and an “Overall Capacity for Sustainability” measure was calculated using the average of the eight domain scores for each health system. Individual pilot-site scores were not calculated for Fam Med or Nursing, since all decisions relevant to implementation and sustainability were made centrally. Pilot-site scores were calculated for each of MD-Group’s five sites, since all relevant decisions were decentralized. Regional raters compared item, domain, and overall scores, and discussed discrepancies and observed examples to justify ratings and reach agreement. Scores from each pair’s work were discussed within each regional academic team in order to reach consensus on scores for their health system. Each regional academic team then presented and justified their ratings during a conference call attended by the three teams. In a few instances, recommendations were made to improve cross-site interpretation and consistency in scoring the domains across all rating teams, using the PSAT definitions as a theory-informed framework and taxonomy to guide interpretation [20]. After all sustainability scores were finalized, the three regional academic teams re-convened to compare differences and similarities in scores across the three health systems as well as challenges in scoring the PSAT. Statistical analyses to compare variation in health system PSAT scores were not performed given the small sample and potential for measurement error. Instead, the numerical scores provided a means of qualitatively comparing relative performance on individual domains and to describe sets of domains observed (or not observed) in sustaining sites and nonsustaining sites. Completing and discussing the PSAT also illuminated variation in domain relevance across systems. Although all domains were relevant to Nursing, the MD-Group and Fam Med raters were unable to score the items within the domains of Partnerships and Communications due to their perceived lack of relevance to those systems. As a public health system, Nursing was accountable to numerous community partners and required to garner public support for their services—activities that were uncommon for the other two systems. Thus, these two domains were dropped from the cross-site comparison and the Overall Capacity for Sustainability scores were recalculated based solely on the remaining six domains. (See Table 2 for PSAT scores.) Table 2 Program Sustainability Assessment Tool (PSAT) domain scores by healthcare system Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n aSystem Scores represent the average of the domain item scores for each healthcare system (scale of 1–7), with scores >3.5 (i.e. the midpoint on the PSAT rating scale) interpreted as relatively “strong.” bIndividual Clinic Scores were calculated separately for the five MD-Group clinics, since implementation and sustainability decisions were made at the clinic level. cOverall Capacity for Sustainability scores represent the averages of all average domain scores. dStatus reported by systems during 12-month follow-up call (y = yes; n = no). View Large Table 2 Program Sustainability Assessment Tool (PSAT) domain scores by healthcare system Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n Fam Med Nursing MD-Group PSAT domains System scores System scoresa Individual clinic scoresb 1 2 3 4 5 Environmental support 5.8 5.4 4.6 5.4 2.8 2.8 2.6 Funding stability 3.4 5.6 4.8 3.2 1.6 1.2 1.2 Strategic planning 3.6 6.2 4.0 3.8 2.0 1.4 1.4 Program evaluation 6.5 4.8 3.6 2.8 2.2 2.0 1.4 Program adaptation 6.4 4.6 5.6 5.0 2.4 2.0 2.2 Organizational capacity 4.8 6.0 5.6 5.0 2.8 2.2 2.2 Overall capacity for sustainabilityc 5.1 5.4 4.7 4.2 2.3 1.9 1.8 Alcohol SBI sustainedd y y y n n n n aSystem Scores represent the average of the domain item scores for each healthcare system (scale of 1–7), with scores >3.5 (i.e. the midpoint on the PSAT rating scale) interpreted as relatively “strong.” bIndividual Clinic Scores were calculated separately for the five MD-Group clinics, since implementation and sustainability decisions were made at the clinic level. cOverall Capacity for Sustainability scores represent the averages of all average domain scores. dStatus reported by systems during 12-month follow-up call (y = yes; n = no). View Large Twelve-month postfunding follow-up call After the PSAT data were analyzed and interpreted with regard to sustainability capacity, academic team partners contacted each healthcare system to confirm if alcohol SBI was sustained. One leader at Nursing, two at Fam Med, and one at each of the five MD-Group clinics were asked: (a) if their practice continues to screen all patients for alcohol misuse and conduct brief interventions or make referrals, as needed; (b) whether they were still using the specific screening tool that they were trained on; and (c) what contributed to continuing/discontinuing alcohol SBI at their organization? Responses to these semistructured questions were summarized and answers to question 1 were dichotomized as either yes (y) or no (n) (see Table 2 for sustainability outcomes). Qualitative comparative analysis (QCA) QCA provides a systematic, case-study approach for comparing sets of potentially relevant factors (i.e. conditions), and determining the extent that cases sharing the same conditions also share the same outcome [21]. QCA can be used for small samples and allows for determining potential pathway(s) to success, as well as failure, with regards to achieving the desired outcome [21]. To create each setting’s conditions, PSAT domain scores for Fam Med, Nursing, and MD-Group’s five sites were dichotomized, with scores greater than 3.5 (i.e., the midpoint on the PSAT rating scale) categorized as “strong” and given a value of “1”, and scores equal to or less than 3.5 categorized as “weak” evidence for each domain, and given a value of “0”. Assigning a value of 1 or 0 is interpreted as full inclusion or exclusion in the set (known as “crisp sets”). Although QCA permits the use of “fuzzy sets” (i.e., partial inclusion), the fact that the PSAT domain scale is ordinal, and ratings were based on qualitative data, dichotomizing was justified. Using methods described by Kane and colleagues [21], a table was constructed, with a row for each unique set of conditions (represented by 0 or 1 for each PSAT domain), the number of sites matching those conditions, and the proportion of those sites (from 0.0 to 1.0) that reported sustaining alcohol SBI at 12 months postpilot follow-up (Table 3). This technique helped to illuminate which conditions, by themselves or in combination with others, were likely necessary to sustain alcohol SBI (i.e., the condition was strong in the clinics that sustained and weak in the clinics that did not sustain alcohol SBI); and which condition(s), when present, were likely sufficient to sustain alcohol SBI (i.e., clinics where only a subset of conditions were rated strong, but they still were sustaining alcohol SBI when contacted 12 months following the pilot). Table 3 Qualitative Comparative Analysis of setting conditions and potential relationship to sustainability outcomes. Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 aConditions: A = Fam Med; B = Nursing; MD-Group Clinic 1; C = MD-Group Clinic 2; D = MD-Group Clinics 3–5. bDomain definitions: Env. Supp. (Environmental Support), champions, leadership, public support; Funding Stability, stable and sustained program funding; Org.Cap. (Organizational Capacity), program is well integrated into operations and supported by leadership, adequate staff, systems; Prog. Eval. (Program Evaluation), reports short-term and intermediate outcomes, results inform planning, monitor fidelity to protocols, demonstrate success; Prog. Adapt. (Program Adaptation), program periodically reviews the evidence base, adapts to assure continued effectiveness; and Strategic Planning, plans for future resources and financial needs; program goals and roles are understood by all stakeholders. 1 = strong; 0 = weak evidence for that domain. cSites (n) totaled 7; (Nursing = 1, Fam Med = 1 and MD-Group = 5, denoted as Clinics 1–5). This total reflects the number of sites with the decision-making authority to sustain or not sustain Alcohol SBI. View Large Table 3 Qualitative Comparative Analysis of setting conditions and potential relationship to sustainability outcomes. Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 Conditionsa Domainsb Env. Supp. Funding stability Org. Cap. Prog. Eval. Prog. Adapt. Strategic planning Sitesc (n) Proportion of sites sustaining alcohol SBI A 1 0 1 1 1 1 1 1.0 B 1 1 1 1 1 1 2 1.0 C 1 0 1 0 1 1 1 0.0 D 0 0 0 0 0 0 3 0.0 aConditions: A = Fam Med; B = Nursing; MD-Group Clinic 1; C = MD-Group Clinic 2; D = MD-Group Clinics 3–5. bDomain definitions: Env. Supp. (Environmental Support), champions, leadership, public support; Funding Stability, stable and sustained program funding; Org.Cap. (Organizational Capacity), program is well integrated into operations and supported by leadership, adequate staff, systems; Prog. Eval. (Program Evaluation), reports short-term and intermediate outcomes, results inform planning, monitor fidelity to protocols, demonstrate success; Prog. Adapt. (Program Adaptation), program periodically reviews the evidence base, adapts to assure continued effectiveness; and Strategic Planning, plans for future resources and financial needs; program goals and roles are understood by all stakeholders. 1 = strong; 0 = weak evidence for that domain. cSites (n) totaled 7; (Nursing = 1, Fam Med = 1 and MD-Group = 5, denoted as Clinics 1–5). This total reflects the number of sites with the decision-making authority to sustain or not sustain Alcohol SBI. View Large RESULTS Conditions supporting sustainability Twelve-month postfunding follow-up calls confirmed sustainment of alcohol SBI as implemented for Fam Med, Nursing, and one MD-Group site. Sustaining sites were scored as strong on at least five of the six PSAT domains: Environmental Support, Program Adaptation, Organizational Capacity, Program Evaluation, and Strategic Planning (Table 2). The consistency in strengths across the two Fam Med and three Nursing sites was likely due to the involvement of centralized leaders and internal champions in the planning, piloting, and sustainability activities. The expertise of Fam Med’s lead project personnel, who were both providers within the Fam Med system and researchers in the field of alcohol misuse, fostered routine review of the science, which was then used in conjunction with findings from their internal quality assurance processes to update trainings and make adaptations that enhanced the overall feasibility and effectiveness of procedures. Fam Med incorporated alcohol SBI into its standard operating procedures through its training, quality improvement activities, and electronic health record (EHR) integration. Despite this success in integrating alcohol SBI into its practice workflows and EHR, Fam Med relayed some uncertainty regarding continuing training, performance monitoring, and billing at the end of the pilot. Also, a change in Fam Med’s leadership affected program support during the transitional period, highlighting a potential threat to sustaining alcohol SBI and indicating a need to engage in more strategic planning. Nursing planning team members drafted a written policy and procedure during the initial planning process, and presented it to Nursing leadership for approval. The intention was to refine the draft, throughout the implementation pilot, as adaptations were made. Nurses at the pilot sites were informed that their input would help shape the final policy and procedure prior to its dissemination, thereby improving its integration into operations, and ultimate acceptance as standardized practice. Drafting, piloting, and refining the written policy is evidence of Nursing’s willingness to make adaptations that improved its Organizational Capacity for sustaining alcohol SBI. Taking the time to assure the policy and protocols were feasible and acceptable to providers likely contributed to Nursing routinizing alcohol SBI, despite provider turnover, changes in nurse managers, and retirement of the administrative leader during the pilot period. Towards the end of the pilot, Nursing disseminated alcohol SBI statewide, across all clinics and itinerant services. Alternatively, MD-Group’s decentralized structure likely contributed to the variation observed across the five pilot sites, since operational decisions were made at the practice level, thereby limiting MD-Group’s capacity to provide the support necessary to sustain alcohol SBI, such as continued staff training and performance monitoring. Two sites (Clinic 1 and Clinic 2) were scored strong on four domains, but only Clinic 1, which added standardized screening questions as required fields within its EHR’s health history form, sustained its alcohol SBI protocols. Funding Stability may also have been a contributing factor, given that Clinic 1, unlike the other MD-Group sites, was not dependent on billing or grants to sustain its activities. Use of process evaluation data informed meaningful program adaptation Our qualitative comparative analysis (Table 3) identified only one factor that was consistently strong for all sustaining sites and consistently weak for all nonsustaining sites—Program Evaluation. During the implementation pilot, all regional teams requested monthly data to track screening rates at the system and pilot site levels, and monitor, to the extent possible, fidelity to alcohol SBI protocols. For the nonsustaining sites, data extraction challenges were pervasive and limited MD-Group’s ability to monitor screening and follow-up services with sufficient regularity to identify issues or areas in need of improvement. For example, three of the sites did not document screening directly in the health records, thereby relying on paper forms and manual data entry. Clinic managers reported that it was too time intensive to track and report these data. For the sustaining sites, modifications were made to the patient database (Nursing) or EHR (Fam Med and MD-Group’s Clinic 1), providing a systematic way to document patient screening results and extract data. For these sites, the use of data for reporting progress on health indicators was valued. For example, as a student health center, Clinic 1, which participated in the American College Health Association-National College Health Assessment (ACHA-NCHA), expressed interest in comparing their alcohol screening results to ACHA-NCHA outcomes. Despite indications that sites rated as strong in Program Evaluation were successful at sustaining alcohol SBI and sites rated as weak were unsuccessful at sustaining, we cannot assume that engaging in data collection and reporting outcomes is sufficient for sustaining change. We observed that the systems using fidelity indicators that they monitored over time could iteratively identify issues and make meaningful adaptations that increased their organizational capacity to sustain change. For example, Fam Med and MD-Group’s Clinic 1 used their EHR to track screening rates and monitor compliance with scoring intervention, and documentation protocols, that were then addressed by system leadership. Although it lacked an EHR, Nursing’s database was modified to include entry of screenings completed or deferred and positive scores, from which a monthly screening report was produced. The report indicated a steep decline in screening rates for one of its pilot sites about 4 months after implementation, indicating a lack of adherence to the draft policy and procedures. In addition, feedback collected by academic project staff during scheduled semistructured follow-up calls revealed that nurses at all pilot sites lacked confidence in conducting brief interventions. Skills-focused booster training was provided at all sites to address both issues. Conversely, four MD-Group sites lacked an electronic mechanism for monitoring and reporting data. Without a systematic way to hold providers and staff accountable to established protocols, data-driven adaptations to improve feasibility or fidelity were lacking. Hence, the inability or unwillingness to engage in program evaluation, or to use the data that are generated to refine processes during implementation, may signal a serious threat to sustainability. Organizational capacity to sustain change not necessarily limited by funding instability Weak funding stability was a common factor at all nonsustaining sites. However, funding was also a concern for one sustaining site, suggesting the compensatory strength of other factors in the face of financial instability. For example, challenges related to funding stability may potentially be overcome by capacity to garner commitment, and by adapting programs to assure that they are efficiently integrated into existing workflows and systems. MD-Group’s lower Funding Stability ratings reflected the inability of three of its five pilot sites to be reimbursed due to alcohol SBI Medicaid billing codes not being activated in Nevada. Another of the sites (Clinic 2), which is funded primarily through grants, also did not sustain alcohol SBI, despite being rated as strong on four of the six domains. MD-Group’s nonsustaining clinics implemented very few adaptations to improve alcohol SBI’s feasibility, and suggestions for process improvement were not successfully integrated into workflows. For example, a recommendation by a clinic manager to create a note template that would enhance provider follow-up on interventions and referrals was not successful since providers were not required to use it. Thus, although the nonsustaining sites acknowledged alcohol SBI’s importance, they did not identify any strategies that would enable them to sustain it without third party reimbursement. On the other hand, Fam Med, a sustaining system, also reported long-term funding challenges, due to its reliance on federal funding to support implementation and personnel costs associated with the training and oversight of the implementation pilot. Although Fam Med embedded the appropriate billing codes in the EHR, its clinical staff and project personnel identified barriers related to reimbursement during the pilot. One barrier was the inability to code for alcohol SBI and a problem-oriented visit on the same day. Despite this serious threat to sustainability, Fam Med continues to provide alcohol SBI, largely due to its success in the other domains, which expanded their organizational capacity. Thus, in the presence of other strengths, a lack of Funding Stability does not destine a setting for failure. Lessons from health systems on sustainability Twelve-month postfunding follow-up interviews with leaders at each setting confirmed their success in sustaining alcohol SBI a full year after the pilot projects ended, and provided an opportunity to further understand the PSAT results. Their answers are summarized below. Fam Med reported that both clinics in their health system continue to screen all patients ages 18 and older for alcohol misuse every 6 months and provide a brief intervention or referral to treatment, as indicated by the screening results. According to the Fam Med healthcare providers, they have continued alcohol SBI at their system due, in large part, to the belief that this type of evidence-based prevention strategy should be standard practice throughout primary care and in related healthcare encounters. One provider said, “Incorporating the process into the EHR, including using health reminders and prompts, was key to standardizing this practice in our clinics.” They note that in an underserved community, such as theirs, it is important to educate patients about risky health behaviors and assist them with seeking outside treatment. Nursing’s section chief credited a number of factors to the overall sustainment of their alcohol SBI program, many of which confirmed PSAT identified strengths. First, she reported that engaging staff in planning, and piloting alcohol SBI prior to system-wide dissemination, allowed them to identify glitches, improve their procedures, and create tools and reports that helped support the implementation process. Second, ongoing standardization and consistency among all staff was achieved by adapting alcohol SBI training modules that could be accessed through their existing electronic learning platform, completion of which is required for all new providers. Last, she described how applying lessons learned from implementing other screenings helped make this one work, saying, “We knew what worked for our DV [Domestic Violence] screening and how that was implemented.” Of the five pilot clinics within MD-Group only Clinic 1, the setting that integrated the screening tool into their EHR system and that was funded through student fees (i.e., not dependent on grants or third party reimbursement), reported sustaining a universal approach to screening and intervening for alcohol misuse, using the screening questions and tools adopted during the implementation pilot. They reported that having the screening tool in the EHR was vital to facilitating the process. The other sites reported that they screen for alcohol sporadically and cited “time factors” as the main barrier to sustaining universal alcohol SBI. DISCUSSION Organizational capacity to sustain change relies on multiple factors that, taken together, contribute to integrating evidence-based interventions into practice. We found that the sites that used their process data to inform adaptations that enhanced the feasibility and fidelity of alcohol SBI, also enhanced their capacity to keep it going after the pilot study was completed. Research describing the barriers and challenges for implementing practice change is plentiful; however, less attention has been given to specifying factors that influence whether changes will be sustained after implementation. Facilitators to practice change, such as provider knowledge and skills training, may themselves be unsustainable if the desired provider behavior change is not supported by organizational infrastructure or protocols [12, 17, 22, 23]. This paper used a sustainability framework and assessment tool and QCA to analyze and compare three regional healthcare systems on six domains that have been associated with sustaining public health programs. The tool provided a common empirical framework and taxonomy from which we used qualitative methods to compare factors that may be necessary and those that may be sufficient to achieving sustainability. While implementation approaches across the sites were similar, a few differences in setting conditions were observed that may have contributed to sustainability success. First, all of the nonsustaining systems were rated low on Funding Stability, a barrier to implementation and sustained change that is frequently reported in the alcohol SBI literature [24]. However, Funding Stability was also a concern in one of the sustaining systems, suggesting that their efforts during implementation to fully imbed alcohol SBI into their practice priorities and workflows enhanced their Organizational Capacity to sustain the change [25, 26]. Second, the practice change literature emphasizes the importance of using data as a process improvement strategy to manage change [27], yet the importance of ongoing program evaluation is seldom identified as a barrier to program sustainability. In our study, the domain of Program Evaluation was a key difference between the sites that sustained alcohol SBI and those that did not. Although several sites used electronic databases or EHRs to cue providers to screen patients, the extent that system-level data were used to monitor, report on, and make program adaptations that improved process fidelity varied. It is interesting to note that Nursing and Fam Med used a paper screening form that involved later data entry into the patient data base or EHR. This did not appear to threaten sustainability, however, given this modality matched their other health risk screenings. What was most important was having a documentation and data extraction protocol to monitor process fidelity. Identifying even one metric that could be monitored monthly, such as percent of eligible patients or visits screened, along with soliciting staff feedback and recommendations to refine protocols, provides a means of identifying problems and improving the logistical fit and sustainment of the intervention within a practice [28–30]. Third, a centralized leadership structure, where decision-making authority for the whole system was concentrated at the top, was associated with sustainability among sites in this study, and decentralized leadership, where decision-making authority to make changes to business operations was held by individual clinic managers, was identified as a barrier to sustainability. Although a study of the influence of organizational dynamics on capacity for implementing sustained practice change is beyond the scope of this paper [31], the ability to achieve standardization and consistency of implementation within a centralized structure may or may not outweigh the importance of achieving the local-level buy-in required by a decentralized structure [32]. It is noteworthy that Nursing and Fam Med’s centralized leadership involved clinic managers and staff in the planning and piloting process of the implementation, promoting both standardization and buy-in. Such collaborative decision-making practices, which foster participation and provide opportunities to provide input, have been strongly associated with sustainability [25]. Decentralized systems may likewise expand their capacity for practice improvement by collaborating with other providers to combine resources and strategies [33]. Thus, proactively compensating for the limitations that may be inherent within centralized or decentralized systems could improve both standardization and acceptance of the practice change. Finally, our follow-up calls with the health system leaders, a year after the project was completed, confirmed that several key PSAT factors were also identified by the health systems as contributing to sustaining alcohol SBI. In addition, health systems mentioned three important factors that may not have been readily observed by external raters: (a) utilizing internal knowledge of what has worked (or not worked) historically with regard to implementing new services may enhance program implementation and proactively address issues that will improve the likelihood of sustainability [34]; (b) acknowledging that alcohol SBI addresses a gap in service provision may improve staff acceptance; and (c) a fundamental belief that providing universal alcohol SBI is aligned with the practice’s commitment to provide appropriate, evidence-based patient care may override deficits in other domains. Although the use of a standardized framework strengthened our ability to focus on domains identified in the literature as important to sustainability, two of the PSAT domains were not considered relevant to the nonpublic health primary care sites, a limitation of using an instrument designed for public health. In addition, the full use of QCA to compare the extent that diverse sites with similar conditions share the same outcome was limited by our sample size and number of domains. With only seven cases that potentially varied on six different dimensions, not all possible combinations of factors were observed. So, although our findings did identify one domain that may be necessary for sustainability (i.e., Program Evaluation), more research on the potential pathways to sustained change is required. Also, our decision to dichotomize the domains may have obscured the potential influence of domains that were somewhat or partially present, as well as their different levels of importance within each system. Our study provided a retrospective assessment conducted by program facilitators as opposed to the healthcare systems themselves. Although raters were involved in most aspects of implementation and had documented observations and feedback provided by the health system and clinic staff throughout the study, comparing their ratings with healthcare system staff would strengthen our findings. On the other hand, our 12-month, post-pilot follow-up calls to system leaders confirmed sustainability outcomes and provided a valuable organizational perspective. CONCLUSIONS AND RECOMMENDATIONS The ability of health systems and implementation researchers to assess and address threats to sustaining practice change as part of implementation planning will strengthen the likelihood that the reach and quality of newly adopted practices are sustained. Consistent with the notion that all healthcare settings are unique, the pathways for successfully sustaining practice change do not appear to conform to one set of conditions. Understanding the domains that contribute to success or failure, and identifying ways to remove the threats or mitigate their influence by strengthening other areas, broadens the potential for success. We recommend that future research explores prospective assessment of the relevant sustainability domains by healthcare systems to determine whether directly addressing potential threats to sustainability (e.g., locating funding sources, shoring up program evaluation capabilities/requirements), or strengthening other domains to compensate, (e.g., adapting programs to assure they are well integrated into workflows and practice standards) results in sustained change. Research that replicates and expands on our results, using a larger sample, would further define the conditions that are necessary and/or sufficient for successfully sustaining practice change. In addition, mediator/moderator analyses to better understand the pathways leading to sustainability would enhance our ability to assess and intervene more precisely. Finally, we recommend that funders and policymakers consider the importance of providing start-up funding, technical assistance, and practical quality indicators that can be used to sustain quality processes and outcomes [19]. Acknowledgments This publication was supported by funding from the U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, cooperative agreement numbers U84DD000882, U84DD000886, and U84DD000888, funding opportunity DD11-1107. The findings and conclusions in this publication are those of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention. Compliance with Ethical Standards Conflict of interest: D.K. King, S.J. Gonzalez, J. Hartje, B.L. Hanson, C. Edney, H. Snell, R.J. Zoorob, and N.A. Roget have no conflict of interest to report. References 1. Minnier TE . How To Build Sustainability Into the Innovation Process ; 2014 . Available at https://innovations.ahrq.gov/perspectives/how-build-sustainability-innovation-process. Accessibility verified March 24, 2014 . 2. Fleiszer AR , Semenic SE , Ritchie JA , Richer MC , Denis JL . The sustainability of healthcare innovations: a concept analysis . J Adv Nurs . 2015 ; 71 ( 7 ): 1484 – 1498 . Google Scholar Crossref Search ADS PubMed 3. Pagoto SL , Spring B , Coups EJ , Mulvaney S , Coutu MF , Ozakinci G . Barriers and facilitators of evidence-based practice perceived by behavioral science health professionals . J Clin Psychol . 2007 ; 63 ( 7 ): 695 – 705 . Google Scholar Crossref Search ADS PubMed 4. Tabak RG , Khoong EC , Chambers DA , Brownson RC . Bridging research and practice: models for dissemination and implementation research . Am J Prev Med . 2012 ; 43 ( 3 ): 337 – 350 . Google Scholar Crossref Search ADS PubMed 5. Johnson K , Hays C , Center H , Daley C . Building capacity and sustainable prevention innovations: a sustainability planning model . Eval Program Plann . 2004 ; 27 ( 2 ): 135 – 149 . Google Scholar Crossref Search ADS 6. Brown PM , Cameron LD , Ramondt S . Sustainability of behavioral interventions: beyond cost-effectiveness analysis . Int J Behav Med . 2015 ; 22 ( 3 ): 425 – 433 . Google Scholar Crossref Search ADS PubMed 7. Fleiszer AR , Semenic SE , Ritchie JA , Richer MC , Denis JL . An organizational perspective on the long-term sustainability of a nursing best practice guidelines program: a case study . BMC Health Serv Res . 2015 ; 15 : 535 . Google Scholar Crossref Search ADS PubMed 8. Shediac-Rizkallah MC , Bone LR . Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy . Health Educ Res . 1998 ; 13 ( 1 ): 87 – 108 . Google Scholar Crossref Search ADS PubMed 9. Scheirer MA , Dearing JW . An agenda for research on the sustainability of public health programs . Am J Public Health . 2011 ; 101 ( 11 ): 2059 – 2067 . Google Scholar Crossref Search ADS PubMed 10. Kaner EF , Dickinson HO , Beyer F , et al. The effectiveness of brief alcohol interventions in primary care settings: a systematic review . Drug Alcohol Rev . 2009 ; 28 ( 3 ): 301 – 323 . Google Scholar Crossref Search ADS PubMed 11. Armstrong N , Herbert G , Brewster L . Contextual barriers to implementation in primary care: an ethnographic study of a programme to improve chronic kidney disease care . Fam Pract . 2016 ; 33 ( 4 ): 426 – 431 . Google Scholar Crossref Search ADS PubMed 12. Hunter SB , Han B , Slaughter ME , Godley SH , Garner BR . Associations between implementation characteristics and evidence-based practice sustainment: a study of the Adolescent Community Reinforcement Approach . Implement Sci . 2015 ; 10 : 173 . Google Scholar Crossref Search ADS PubMed 13. Scheirer MA . Linking sustainability research to intervention types . Am J Public Health . 2013 ; 103 ( 4 ): e73 – e80 . Google Scholar Crossref Search ADS PubMed 14. Hassmiller Lich K , Frerichs L , Fishbein D , Bobashev G , Pentz MA . Translating research into prevention of high-risk behaviors in the presence of complex systems: definitions and systems frameworks . Transl Behav Med . 2016 ; 6 ( 1 ): 17 – 31 . Google Scholar Crossref Search ADS PubMed 15. Moyer VA ; Preventive Services Task Force . Screening and behavioral counseling interventions in primary care to reduce alcohol misuse: U.S. preventive services task force recommendation statement . Ann Intern Med . 2013 ; 159 ( 3 ): 210 – 218 . Google Scholar PubMed 16. Luke DA , Calhoun A , Robichaux CB , Elliott MB , Moreland-Russell S . The Program Sustainability Assessment Tool: a new instrument for public health programs . Prev Chronic Dis . 2014 ; 11 : 130184 . Google Scholar Crossref Search ADS PubMed 17. Calhoun A , Mainor A , Moreland-Russell S , Maier RC , Brossart L , Luke DA . Using the Program Sustainability Assessment Tool to assess and plan for sustainability . Prev Chronic Dis . 2014 ; 11 : 130185 . Google Scholar Crossref Search ADS PubMed 18. Wiltsey Stirman S , Kimberly J , Cook N , Calloway A , Castro F , Charns M . The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research . Implement Sci . 2012 ; 7 : 17 . Google Scholar Crossref Search ADS PubMed 19. Aldridge WA II , Boothroyd RI , Fleming WO , et al. Transforming community prevention systems for sustained impact: embedding active implementation and scaling functions . Transl Behav Med . 2016 ; 6 ( 1 ): 135 – 144 . Google Scholar Crossref Search ADS PubMed 20. Bradley EH , Curry LA , Devers KJ . Qualitative data analysis for health services research: developing taxonomy, themes, and theory . Health Serv Res . 2007 ; 42 ( 4 ): 1758 – 1772 . Google Scholar Crossref Search ADS PubMed 21. Kane H , Lewis MA , Williams PA , Kahwati LC . Using qualitative comparative analysis to understand and quantify translation and implementation . Transl Behav Med . 2014 ; 4 ( 2 ): 201 – 208 . Google Scholar Crossref Search ADS PubMed 22. Virani T , Lemieux-Charles L , Davis DA , Berta W . Sustaining change: once evidence-based practices are transferred, what then ? Healthc Q . 2009 ; 12 ( 1 ): 89 – 96, 2 . Google Scholar Crossref Search ADS PubMed 23. Seale JP , Velasquez MM , Johnson JA , et al. Skills-based residency training in alcohol screening and brief intervention: results from the Georgia-Texas “Improving Brief Intervention” Project . Subst Abus . 2012 ; 33 ( 3 ): 261 – 271 . Google Scholar Crossref Search ADS PubMed 24. Bernstein E , Topp D , Shaw E , et al. A preliminary report of knowledge translation: lessons from taking screening and brief intervention techniques from the research setting into regional systems of care . Acad Emerg Med . 2009 ; 16 ( 11 ): 1225 – 1233 . Google Scholar Crossref Search ADS PubMed 25. Durlak JA , DuPre EP . Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation . Am J Community Psychol . 2008 ; 41 ( 3–4 ): 327 – 350 . Google Scholar Crossref Search ADS PubMed 26. Fixsen DL , Naoom SF , Blase K , Friedman RM , Wallace F. Implementation research: a synthesis of the literature . Tampa, FL : Louis de la Parte FL Mental Health Inst .; 2005 . 27. Clark KD , Miller BF , Green LA , de Gruy FV , Davis M , Cohen DJ . Implementation of behavioral health interventions in real world scenarios: Managing complex change . Fam Syst Health . 2017 ; 35 ( 1 ): 36 – 45 . Google Scholar Crossref Search ADS PubMed 28. Cooper BR , Bumbarger BK , Moore JE . Sustaining evidence-based prevention programs: correlates in a large-scale dissemination initiative . Prev Sci . 2015 ; 16 ( 1 ): 145 – 157 . Google Scholar Crossref Search ADS PubMed 29. Moore JE , Bumbarger BK , Cooper BR . Examining adaptations of evidence-based programs in natural contexts . J Prim Prev . 2013 ; 34 ( 3 ): 147 – 161 . Google Scholar Crossref Search ADS PubMed 30. Elliott DS , Mihalic S . Issues in disseminating and replicating effective prevention programs . Prev Sci . 2004 ; 5 ( 1 ): 47 – 53 . Google Scholar Crossref Search ADS PubMed 31. Harlos K , Tetroe J , Graham ID , Bird M , Robinson N . Mining the management literature for insights into implementing evidence-based change in healthcare . Healthc Policy . 2012 ; 8 ( 1 ): 33 – 48 . Google Scholar PubMed 32. Davis MV , Mahanna E , Joly B , et al. Creating quality improvement culture in public health agencies . Am J Public Health . 2014 ; 104 ( 1 ): e98 – 104 . Google Scholar Crossref Search ADS PubMed 33. Lanter PL , Wolff KB , Johnson LC , Ercolano EM , Kilmer JR , Provost L . Change IS Possible: Reducing High-Risk Drinking Using a Collaborative Improvement Model . J Am Coll Health . 2015 ; 63 ( 5 ): 330 – 336 . Google Scholar Crossref Search ADS PubMed 34. Sackmann SA , Eggenhofer-Rehart PM , Friesl M . Sustainable Change: Long-term efforts toward developing a learning organization . J Appl Behav Sci . 2009 ; 45 ( 4 ): 521 – 549 . Google Scholar Crossref Search ADS © Society of Behavioral Medicine 2018. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Journal

Translational Behavioral MedicineOxford University Press

Published: Sep 8, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off