Background: Sustaining evidence-based interventions (EBIs) is an ongoing challenge for dissemination and implementation science in public health and social services. Characterizing the relationship among human resource capacity within an agency and subsequent population outcomes is an important step to improving our understanding of how EBIs are sustained. Although human resource capacity and population outcomes are theoretically related, examining them over time within real-world experiments is difficult. Simulation approaches, especially agent-based models, offer advantages that complement existing methods. Methods: We used an agent-based model to examine the relationships among human resources, EBI delivery, and population outcomes by simulating provision of an EBI through a hypothetical agency and its staff. We used data from existing studies examining a widely implemented HIV prevention intervention to inform simulation design, calibration, and validity. Once we developed a baseline model, we used the model as a simulated laboratory by systematically varying three human resource variables: the number of staff positions, the staff turnover rate, and timing in training. We tracked the subsequent influence on EBI delivery and the level of population risk over time to describe the overall and dynamic relationships among these variables. Results: Higher overall levels of human resource capacity at an agency (more positions) led to more extensive EBI delivery over time and lowered population risk earlier in time. In simulations representing the typical human resource investments, substantial influences on population risk were visible after approximately 2 years and peaked around 4 years. Conclusions: Human resources, especially staff positions, have an important impact on EBI sustainability and ultimately population health. A minimum level of human resources based on the context (e.g., size of the initial population and characteristics of the EBI) is likely needed for an EBI to have a meaningful impact on population outcomes. Furthermore, this model demonstrates how ABMs may be leveraged to inform research design and assess the impact of EBI sustainability in practice. Keywords: Sustainability, Agent-based modeling, Evidence-based intervention, Human resources, Dissemination and implementation science, Organizational capacity, Systems science * Correspondence: Virginia.firstname.lastname@example.org Center for Public Health Systems Research in the Warren G. Brown School of Social Work, Washington University in St. Louis, Campus Box1196, One Brookings Drive, St. Louis, MO 63130, USA Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. McKay et al. Implementation Science (2018) 13:77 Page 2 of 10 Background grounded in organizational theory for health organizations Evidence-based interventions (EBIs) are intended to help [8, 9] and empirical evidence , and it suggests that the ensure beneficial outcomes for the individuals and com- level of capacity within an organization influences the ser- munities that receive them and have been widely imple- vices it provides by the organization. The organizational mented in public health practice. Dissemination and capacity model incorporates multiple components that implementation in research is dedicated to understanding comprise capacity (e.g., financial, human, physical, and the factors that influence the dissemination and imple- informational resources) and identifies specific variables to mentation of EBIs . Although many EBIs are adopted operationalize and measure each component. Human and initially implemented, adequately delivering and sus- resources are identified as a major, and perhaps the most taining an EBI is an ongoing challenge in practice [2–4]. vital, component of organizational capacity and are opera- Organizational capacity, especially adequate staffing, is tionalized as number of full-time employees, staff know- commonly reported as a contributing factor to whether or ledge and skills, education experience, training, staffing not an EBI is sustained. Without adequate staffing, EBIs configuration, retention/turnover, and compensation. may not be appropriately provided to meet community To complement our use of the organizational capacity need (e.g., insufficiently available or underused) or may be model and its exposition of human resources, Scheirer prematurely abandoned altogether. Furthermore, inad- and Dearing’s conceptual framework of EBI sustainabil- equate EBI delivery may not have demonstrable effects on ity identifies variables for measuring sustainability. This the intended population outcomes. While there is need to model defines sustainability as the final stage of program identify the appropriate level of staffing to bolster the suc- implementation when programs are maintained and be- cessful delivery and sustainability of EBIs in practice, a come integrated into the regular functioning of an particular challenge of empirically examining these issues organization . Both models hypothesize a relation- is the longitudinal and dynamic nature between adequate ship between available human resources, EBI sustainabil- staffing and EBI sustainment [5, 6]. We use agent-based ity, and downstream community benefits. If the EBI is modeling, a computational systems science approach, in no longer benefiting the community as a whole, then it conjunction with existing EBI implementation frameworks may need to be adapted, discontinued, and/or replaced and empirical data, to examine the relationships between to better align with community need. staffing, sustainability, and population health. Challenges to examining EBI sustainability Organizational capacity, EBI sustainability, and population Identifying essential resource components, for example outcomes minimum numbers of staff dedicated to an interven- We frame the relationship between human resources, EBI tion, should ideally be part of assessing EBI delivery, sustainability, and population outcomes with the general sustainment, and intended outcomes. However, exam- premise that the level of human resources influences the ining the relationships between human resources and extent to which organizations deliver and sustain an inter- EBI implementation over extended periods of time is vention, which, in turn, influences how the intervention difficult using common research designs. EBI sustain- impacts the intended population. These relationships are ability occurs over relatively longer time periods than conveyed in Fig. 1. Human resources fall within the earlier implementation phases (e.g., adoption and early conceptual domain of organizational capacity, describing implementation). During the sustainability phase, EBI the resources available to public health organizations to activities ideally persist despite common changes in the deliver essential services and improve population health [6, agency, like staff turnover . Using longitudinal de- 7]. Public health organizations require adequate capacity to signs—following organizations over extended periods of sustain services at appropriate levels and meet community time and collecting large volumes of quantitative data— needs over time. The organizational capacity model devel- is costly and impractical, making capturing relation- oped by Meyer, Davis, and Mays provides a framework for ships between human resources and EBI sustainability organizational capacity and services . This model is difficult . Additionally, researchers often only assess the influence of EBI delivery on direct recipients of services but cannot capture the impact of EBI delivery for entire communities. Qualitative designs are used to examine organizational capacity and EBI implementa- tion because they offer the necessary detail that is required to explain the relationships that may influence Fig. 1 ABM variables using organizational capacity model developed EBI sustainability [14, 15], but the ability to generalize by Meyer et al.  and integrated with EBI sustainability model by results from these studies across different contexts is Scheirer and Dearing  limited. McKay et al. Implementation Science (2018) 13:77 Page 3 of 10 Agent-based modeling to examine sustainability Data sources Agent-based modeling (ABM) is one approach that offers We used the organizational capacity model developed by several advantages in examining EBI sustainability. ABMs Meyer, Davis, and Mays; the EBI sustainability model by model the behavior of heterogeneous individuals (i.e., Scheirer and Dearing [7, 11](seeFig. 1); and data from agents) and the interaction between them in an environ- existing studies focusing on the EBI, RESPECT, to inform ment. The aggregated individual behaviors of agents illu- our ABM (see Table 1). RESPECT is a brief, minate the dynamics of the larger social structures they evidence-based counseling and testing intervention that both comprise and create . Simulations can be con- aims to reduce sexually transmitted infections and HIV by structed using existing empirical data and then manipu- reducing high-risk sexual behaviors that make infection lated to produce alternative outcomes. These models can transmission more likely [25, 26]. As originally designed, also inform hypothesis generation, guide future data collec- RESPECT is delivered in a two-session format where cli- tion, and inform theory development. In this way, ABM ents identify risk behaviors and develop a plan for reducing complements both qualitative and quantitative research identified behaviors in the first counseling session. In the approaches by incorporating details of context while pro- second session, clients discuss their success or failure in viding a virtual lab to generate plausible explanations and achieving their risk-reduction plan and identify additional descriptions of relationships among variables over a behaviors to help support and improve behavioral changes. lengthy period of time . A small number of ABMs have RESPECT was supported by the Centers for Disease been developed to assess the impact of public health policy Control and Prevention (CDC) and was one of several EBIs and services in communities [18–21], organizational widely implemented for reduction of sexually transmitted behavior , and EBI implementation for disease preven- infections in local settings . tion . These ABMs show promise in their ability to We used published statistics from the original demonstrate the impact of EBIs on population health but RESPECT randomized-control trial [25, 26] and CDC have been underutilized in dissemination and implementa- publications characterizing standard program structure tion research . and protocols  to inform overall model design. We also used data from two RESPECT implementation stud- ies to inform the characterization of human resources The current study and acquire data on RESPECT clients, staff, and agencies The strengths of ABM provide an opportunity to examine [29–32]. Specific variables, descriptions, values, and em- the sustainability of EBIs in public health organizations pirical data sources for each variable used to inform over time. The approach may also offer insights into simulation runs are provided in Table 1. factors and dynamics influencing EBI sustainability that are otherwise difficult to obtain. Guided by the theoretical Model design and development premise that optimal human resources lead to more We used Netlogo 5.2 to develop, calibrate, validate, and sustainable EBIs and greater improvements in population execute the ABM . Each simulation run begins by health, we use ABM to explore the relationship among generating a number of locations where the EBI is deliv- human resources, EBI sustainability, and population ered, a provider population, and a client population. A outcomes to inform theoretical models of organizational screen shot of the simulation view and code are provided capacity and demonstrate the use of ABM to address in Additional file 1. The agency has a maximum number pressing EBI sustainability research questions. (from two to ten) of provider positions available. At the beginning of a simulation run, the agency hires the max- Methods imum number of staff to fill all available positions, which Our ABM models the delivery of an EBI by staff to a we refer to as staff positions. All newly hired staff begin population in a hypothetical agency over a period of years. as untrained. Initially, the model creates an agency with staff hired to The simulation also generates a population for recruit- provide the intervention and the population. As the simu- ment to receive the EBI. There are limited data to sup- lation moves forward in time, staff are trained and deliver port the number of individuals at risk for a sexually the EBI to a population at risk for human immunodefi- transmitted infection in any community. However, crude ciency virus (HIV). As staff leave the agency, the agency estimates of populations at increased risk for HIV, for hires and trains new staff to fill the number of available example, give a very basic sense of the number of indi- positions. We assessed the influence of human resources viduals at risk. For example, men who have sex with by systematically adjusting three key human resource vari- men, who make up 63% of all new HIV cases, make up ables: the number of staff positions at the agency, the rate approximately 2 % of the US population . Presuming of turnover, and the length of time required for training 2 % of the population is at increased risk for HIV, a newly hired staff for EBI delivery. community of approximately 500,000 would have a risk McKay et al. Implementation Science (2018) 13:77 Page 4 of 10 Table 1 Key simulation variables and parameters Variable Description Data source(s) Value 1.1 Loss to follow up (%) The proportion of clients that receive the first session of RESPECT but will not RCT 15 return for the second session 1.2 Risk reduction achieved (%) The proportion of clients that will achieve their risk-reduction step RESPECT case 72 1.3 Size of risk reduction (M; The size of the risk-reduction step achieved by the clients selected from normal RCT; RESPECT case 1; 0.5 SD) distribution 1.4 Clients in a week (N) The number of clients seen by one provider in a week Project RESPECT 15 1.5 Risk decay (%) The proportion of clients that initially achieved their risk-reduction step but RCT 8 experience an increase in risk 1.6 Size of risk decay (M; SD) The size of the risk-reduction step achieved by the clients selected from a normal RCT; RESPECT case 1; 0.5 distribution 1.7 Repeat eligibility criteria The criteria determining whether an individual in the population can participate Project RESPECT 3; < 4 (Months; Risk) in the intervention again Project RESPECT = data from translation of project RESPECT (see references [28–30]). RCT = data from the original RESPET randomized controlled trial (see references [24, 25]). RESPECT case = data from the RESPECT de-adoption study (see reference ) population of 10,000, and the target population size in EBI are randomly selected for recruitment from the our model was held constant at 10,000 individuals. target population and meet with an available provider. The population of potential clients represents the num- Following data from the literature discussed above (see ber of individuals in the population at increased risk for a Table 1), clients participate in the first session if they are sexually transmitted infection or HIV. Given the ongoing new, and a proportion (15%) are lost to follow up and debate around the relationship between risk behaviors and will not return for the second session. A proportion of actual risk of transmission , we represent risk of trans- the clients who return for a second session (72%) will mission (i.e., the likelihood that an individual will get HIV achieve a risk-reduction step, and the individual risk based on their behavior) as an abstract value rather than value for each client is reduced. The size of the imply risk of transmission using specific risk behaviors as risk-reduction value is randomly selected from a normal a proxy (i.e., sex without condoms or intravenous drug distribution with a relatively modest mean and standard use). We selected a Poisson distribution of risk based on deviation. After receiving either session, all clients return existing literature suggesting larger proportions of individ- to the general target population. The simulation loops uals in the USA are at relatively low risk for HIV transmis- through this set of procedures 15 times for each trained sion, while smaller proportions are at relatively high risk and available provider to represent a week’s worth of EBI [36, 37]. We held the mean of the distribution constant at delivery to clients, based on a reported average number 2.5 at the start of a simulation for all simulation runs. We of clients seen per week from RESPECT (see Table 1). capped individual risk values at zero and seven for two At the end of the week, clients that have received both reasons: (1) Individuals cannot exhibit less than zero risk EBI sessions and achieved their risk-reduction step may (i.e., they have no risk) and (2) this places individuals on a be randomly selected to experience behavior decay, and natural risk scale where individuals on the lower end of revert back to a higher risk level (see Table 1). Similar to the scale (i.e., less than three) can be interpreted as a risk reduction, the size of the risk increase is selected low-risk group and individuals on the higher end of the from a normal distribution with a relatively modest scale (i.e., greater than five) can be interpreted as a mean and standard deviation. Clients become eligible to high-risk group. participate in the EBI again after a 3-month period and if they are at exceptionally high risk for infection (i.e., a Simulation runs risk value over 4). Over time, the agency has two basic sets of activities. The first is a set of activities representing regular, daily EBI de- Weekly agency capacity livery to clients by providers, and the second is a set of The simulation runs through a series of procedures to weekly activities assessing and updating available pro- update the agency’s available providers. Again, following viders (i.e., the human resource capacity) at the agency. data from the literature discussed above (see Table 2), all newly hired providers receive training over a number of Regular EBI delivery weeks (two to six). Once a provider is trained, the indi- Based on the availability of trained staff at the agency, vidual is available to provide the EBI to recruited clients clients are recruited from the target population for the during the week. Over time, providers have a probability EBI. Clients who have not yet received both sessions the (0.05 to 0.15) of leaving (i.e., turning over), and when McKay et al. Implementation Science (2018) 13:77 Page 5 of 10 Table 2 Experimental conditions values of the experimental parameters in Table 2. For ex- ample, the simulation was run ten times with two available Variable Description Range Increment staff positions, a 5% yearly turnover rate, and a 2-week Staff The maximum number of possible 2–10 2 positions (N) EBI staff positions at the agency delay from the time a new individual was hired to the time that they were provided training. Analysis of model output Turnover The proportion of providers that 5–15 5 rate (%) will turnover in a year data was performed using R 3.3.2 . Timing in The number of weeks that a provider 2–62 training (N) will be present at the agency before Results being trained in the EBI Outcomes from simulation runs are presented in Figs. 2, 3, and 4. We observed the expected general relationship this happens, a position at the agency then becomes avail- among human resources, EBI implementation, and popula- able and an untrained provider is hired to fill the position. tion risk. Higher levels of human resources (e.g., more staff Each time step in the simulation represents 1 week of EBI positions) resulted in more EBI delivery and greater de- delivery. The simulation ends after the equivalent of creases in population risk in a shorter amount of time. 10 years of agency operations (50 ticks = 1 year). Conversely, lower overall levels of human resources resulted in less EBI delivery and limited influence on popu- Validation lation risk, such that in some configurations, there was no We developed our model using best practices [38, 39]. apparent influence on population risk. Within these condi- All code was reviewed by an independent coder. We val- tions, EBI implementation had little influence on popula- idated a baseline simulation of typical delivery of the EBI tion risk because the rate of delivery and achievement of by using existing data. Baseline input parameters were risk reduction were either less than or equivalent to the the following: five provider positions, a turnover rate combined rate of loss to follow up and behavior decay. from 5 to 10% per year, and training 4 weeks after hired. Model outputs for our baseline model were calibrated to Number of staff positions meet the following specific outcomes for clients after Figure 2 presents a contour plot of simulation runs over 1 year: a 15% loss to follow up, 89% achievement of the full 10 years. The primary human resource variable risk-reduction step, and risk-reduction decay of 8% per represented in the figure is the number of available staff year (see Table 1). positions dedicated to the EBI. The other two human re- source variables, staff turnover and training, are held Analysis constant at mean values, 10% and 4 weeks, respectively. To identify the essential human resources to maximize The y-axis is the mean risk of the population. The x-axis impact on population risk, we conducted a set of experi- is time in years (0–10 years). Each line in the plot is the ments systematically varying the level of human resources. number of staff positions (2–10 positions) and is differ- The simulation was run ten times for each combination of entiated by color. Less saturated colors approaching Fig. 2 Contour map of change in population risk by staff positions over time McKay et al. Implementation Science (2018) 13:77 Page 6 of 10 Fig. 3 Trend graphs of the change in mean population risk and the proportion of the population considered high risk (risk > 5) over time white represent greater change in risk, and more satu- We also examined the influence of staff positions on rated colors approaching black represent less change in EBI delivery over time on those at greatest risk and most risk. The plot demonstrates that simulation runs with in need of intervention. The bottom panel of Fig. 3 two positions show a limited amount of influence on shows the percentage of population at high risk (risk > population risk at any time point. Among simulation 5) on the y-axis for the same numbers of staff positions runs with between four and eight positions, changes in as in the top panel (with staff turnover and training still population risk become visible around 2 years, with constant). Similar to outcomes for the overall population most simulation runs reaching peak change in popula- risk, simulations with only two staff dedicated to the EBI tion risk between 3 and 5 years. Interestingly, there is showed the least influence on the population proportion limited additional benefit in terms of population-level at increased risk. In contrast to the outcomes for the risk reduction among simulation runs with eight to ten mean risk in the overall population, under EBI imple- positions relative to simulation runs with six positions. mentation with more than two staff positions, observ- The top panel in Fig. 3 shows the mean population able benefits in decreasing the proportion of the risk over time for two, four, six, eight, and ten staff posi- population at high risk continued throughout the tions. Here again, the other two human resource vari- 10-year period. Of particular note, we did not observe a ables are held constant at values identical to those in floor effect—rather, individuals at increased risk relative Fig. 2. Like in Fig. 2, we see that most of the largest de- to the rest of the population continued to benefit from creases in risk come between 3 and 5 years into EBI im- the EBI throughout the entire 10-year simulated period. plementation. However, for simulation runs with four staff positions, the largest decrease in risk is seen around Training and turnover 7 years, and for those runs with two providers, the mini- Figure 4 shows the mean population risk (y-axis) over mum risk is observed at 10 years. More staff positions the 10-year period (x-axis), and each panel represents re- result in quicker rates of decrease in risk initially, but sults at a different value of training time or turnover eventually risk levels out, and at 10 years, the mean risk rate. The first set of three panels shows the risk by num- for any number of staff positions centers around 2.3. ber of staff positions for the three unique values of staff McKay et al. Implementation Science (2018) 13:77 Page 7 of 10 Fig. 4 Training delays and turnover rates with mean population risk over time training times (2, 4, and 6 weeks) holding turnover constant were needed to adequately deliver the EBI to an at its middle value of 10%. We see little or no discernable adequate proportion of the target population, especially difference between the simulation run results when training given the likelihood that not all individuals will com- time varies, e.g., the difference in mean risk at 10 years with pletely participate in the EBI (i.e., some will be lost to ten staff members for a 2-week and a 6-week training time follow up) or maintain the benefits of the EBI (i.e., is approximately 0.02. We see similar results as the annual some will reverse achieved behavior changes). This re- turnover rate varies from 5 to 15% in the three panels on flects the reality of contextual factors that influence EBI the right (holding training time constant at 4 weeks). Inves- success and sustainability. It also suggests that organi- tigation of potential multiplicative effects, e.g., in simulation zations attempting to implement EBIs without adequate runs with the maximum training time and maximum turn- capacity may see individuals reap benefits in reduced over rate, similarly showed no substantial impact when risk, but not see benefits in the overall population (i.e., controlling for number of staff positions. reduced mean risk). Our ABM suggests that achieve- ment of population risk reduction is contingent on the Discussion context in which an EBI is implemented, including the We explored the influence of human resources on EBI size of the target population, the level of organizational delivery over time and risk at the population level. Using capacity supported over time, and the characteristics of an ABM based on available empirical evidence, we ob- EBI delivery. served many of the theoretically supported relationships This ABM also supports empirical evidence that among human resources, EBI sustainability, and popula- smaller agencies, or agencies that are only able to dedi- tion impact [7, 11]. We also were able to explore and de- cate a few staff to a particular service, have more diffi- scribe many of the dynamic interactions among these culty sustaining EBIs and achieving population benefits variables that would otherwise be difficult to examine. [29, 41]. While timing in training and turnover had lit- The outcomes from our ABM have several practical im- tle impact on population outcomes relative to the abso- plications for assessing sustainability of EBIs and their lute number of staff, these results should be interpreted impact on population health. with caution. For example, turnover, especially at higher levels, can impact service quality and drive up costs in Human resource, EBIs, and population dynamics staff replacement and training . Moreover, we did As the organizational capacity model suggests , over- not model financial resources, another major compo- all levels of human resources influenced the extent of nent of organizational capacity and a major predictor of EBI sustainability and the impact of the EBI on popula- EBI sustainability [7, 11], which may be a valuable tion risk. As expected, a minimum human resource in- extension of this model for others seeking to examine vestment was necessary for the EBI to have important the dynamic relationships among organizational population benefits. Sufficient numbers of practitioners capacity variables. McKay et al. Implementation Science (2018) 13:77 Page 8 of 10 Assessing EBI sustainability and population benefit quantitative longitudinal designs . Through ABM, we The ABM also has implications for empirical examina- were able to circumvent these limitations, examine EBI tions of human resources, EBI sustainability, and popu- sustainability over a much longer period of time (the lation outcomes. We used the Scheirer and Dearing equivalent of 10 years) than would have possible or feas- framework of EBI sustainability to model EBI mainten- ible using conventional methods, and demonstrate how ance and delivery over time . This model suggests individual-level changes in a public health outcome gives that EBIs should be sustained for a number of years, but rise to overall population-level characteristics. This is sup- only be sustained as long as an EBI continues to benefit ported by our ABM results showing that while the overall to the population. It is difficult to determine when agen- mean population risk may seem to level out over time, the cies, policymakers, and researchers might expect to see proportion of the population at high risk may continue to the benefits of an intervention, when EBIs might reach decrease. More investigations into how EBI sustainability their maximum benefit, who the EBI benefits most, or influences different portions of the population, especially how often to measure its impacts . Our ABM demon- those most at risk, are needed to help justify or rethink strated intervention benefits after at least 2 years, peak- initial resource allocation and timing and to limit the pos- ing after two or more additional years. However, if there sibility that efforts do not exacerbate or create disparities were fewer human resource investments, evidence of in health outcomes. benefits for both the overall and target populations was To address these issues, ABMs build on theory to guide much slower. As such, assessments may need to be con- assumptions for future models as well as for informing ducted at multiple years post implementation to assess longitudinal research designs. ABMs could be used to esti- continued benefit for the target population. We did not mate which variables are likely most influential on the observe drastic differences in population benefit within a outcomes of interest and, as such, should be a priority for single year, suggesting the ideal interval for assessing RE- prospective data collection. For example, an emergent re- SPECT or a similar EBI's impact may be every 2 to sult of our ABM is evidence that there may not be add- 3 years. While these estimates are likely highly itional benefit if the number of staff positions exceeds dependent on the specific EBI and context of delivery, population need for an intervention. This may be counter- our results suggest that in assessing sustainability, mul- intuitive, since the natural inclination is to believe that tiple year intervals may be most appropriate. This is in simply having more resources to address a public health contrast to other phases of EBI implementation, like ini- problem is better. Although an excess of investment rarely tial adoption, which often unfold in a series of months. seems to be the case in reality given that public health ef- Furthermore, our ABM suggests that overall population forts are often underfunded, assessing optimal levels of benefit and target population benefits might become evi- support can ensure more effective use of resources. dent as well as differential. For example, individuals at Additionally, ABMs could provide insight into key time greatest risk continued to benefit from the EBI over the points when effects of organizational-, program-, and entire 10 years over the simulation run, suggesting that ra- individual-level variables will likely become evident in ther than stopping an intervention entirely, it may be population outcomes. The value of ABMs for implemen- more appropriate to narrow eligibility criteria for an inter- tation research will be enhanced through the use of imple- vention to a more targeted portion of the population or mentation frameworks to inform model development, adapt the intervention once the intervention has saturated identify key variables and relationships modeled, and a current population to be more resource-efficient. This justify model assumptions. supports the current strategy of many public health ap- ABMs may also offer several practical benefits for proaches, especially in HIV prevention, which targets practitioners and policy makers. There are increasing those at highest risk, even among traditionally at-risk numbers of EBIs available for any particular health issue groups like men who have sex with men . . It is often difficult to assess how any EBI will fit with an agency, agency resources, community, or population The potential value of ABM for EBI sustainability . ABMs simulating EBIs could help agencies assess, This study demonstrates how ABM can be a useful strat- select, and adapt the most appropriate EBI to fit the egy in dissemination and implementation research to ad- local context. Similarly, ABMs may help policy makers dress the complex interactions among factors influencing compare how different EBIs or a combination of EBIs EBI sustainability and intended population outcomes. delivered at the local level influence population-level Two commonly cited methodological challenges associ- outcomes. For example, we modeled an HIV counseling ated with examining EBI sustainability are (1) the amount intervention focused on reducing risk through behavior of time needed to adequately follow EBIs and indications change. However, there are numerous approaches to of population benefit and (2) the volume of data needed HIV prevention including behavioral, biomedical, and to adequately fit statistical hierarchical models as part of structural EBIs that address different aspects of HIV risk. McKay et al. Implementation Science (2018) 13:77 Page 9 of 10 Many have debated the relative success of one type of Acknowledgements We would like to thank Heather Bird Jackson, PhD, for aiding with simulation HIV intervention approach over another (e.g., the success programing and reviewing simulation code. We would also like to thank of behavioral interventions compared to the success of other members of the dissertation committee, Joseph A. Catania, and Brian biomedical interventions) [44–46]. As others are begin- Flay for their comments on the early versions of this manuscript. ning to demonstrate , ABMs, which model heteroge- Funding neous systems (e.g., different types of interventions), could This work was supported in part by the Ruth E. Warnke Graduate Fellowship be used to view how a collection of EBIs influences health awarded by the College of Public Health and Human Sciences at Oregon State University and the Provost’s Distinguished Fellowship awarded by outcomes. Oregon State University to Virginia Mckay. NIMH RO1 MH085502-01 awarded to M. Margaret Dolcini, NIMH T32 MH019960. Limitations Availability of data and materials ABM models represent features of real-world systems, but The datasets used to inform the current study are available from the first author on a reasonable request. The programming code from the agent- like models used in other scientific fields, they inherently based model is available as an attachment. reduce the complexity of such systems. Thus, the re- searcher must identify the essential components of the sys- Authors’ contributions All authors contributed to the model design and analysis. All authors tem and make simplifying assumptions about relationships contributed in writing all sections of the manuscript. All authors read and that are related to the processes in question . For ex- approved the final manuscript. ample, we modeled a narrow set of organizational capacity Ethics approval and consent to participate variables and a relatively static client population that did Not Applicable. not fluctuate in size over time. In reality, client populations change over time. Thus, our simulation does not incorpor- Competing interests The authors declare that they have no competing interests. ate the types of population dynamics that would naturally occur in a real context. We have focused on illustrating a Publisher’sNote specific set of relationships using a rather generic model of Springer Nature remains neutral with regard to jurisdictional claims in EBI implementation. While we extensively used theoretical published maps and institutional affiliations. and empirical literature to identify essential components, Author details guide the simulation design, and make simplifying assump- Center for Public Health Systems Research in the Warren G. Brown School tions, we recommend conservative application of the re- of Social Work, Washington University in St. Louis, Campus Box1196, One sults demonstrated in this model to specific circumstances Brookings Drive, St. Louis, MO 63130, USA. Department of Anthropology, Case Western Reserve University, Mather Memorial Room 238 11220, and replication of our findings. Including more variables Bellflower Road, Cleveland, OH 44106-7125, USA. School of Social and relevant to the sustainability of EBIs, such as funding to Behavioral Health Sciences, College of Public Health and Human Sciences, support intervention delivery or additional population dy- Oregon State University, Hallie E. Ford Center for Health Children and Families, 2631 SW Campus Way, Corvallis, OR 97331, USA. namics, may impact the outcomes from our study and would be valuable extensions of the model. Received: 15 August 2017 Accepted: 21 May 2018 Conclusions References Adequate human resources, especially sufficient num- 1. Ogden T, Fixsen DL. Implementation science: a brief overview and a look ahead. Z Psychol. 2014;222(1):4–11. bers of trained staff, are a key influence on EBI sustain- 2. Wiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The ability and population outcomes. However, the sustainability of new programs and innovations: a review of the empirical relationships among human resources, EBI sustainability, literature and recommendations for future research. Implement Sci. 2012;7:17. 3. Hodge LM, Turner KM. Sustained implementation of evidence-based and population outcomes are dynamic and complex. programs in disadvantaged communities: a conceptual framework of Agent-based modeling helps to demonstrate these dy- supporting factors. Am J Community Psychol. 2016;58(1–2):192–210. namic relationships and is an innovative tool for explor- 4. Brownson RC, Allen P, Jacob RR, Harris JK, Duggan K, Hipp PR, et al. Understanding mis-implementation in public health practice. Am J Prev ing aspects of organizational capacity, EBI sustainability, Med. 2015;48(5):543–51. and population health in the future. 5. Proctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10(1):88. Additional file 6. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Additional file 1: A screen shot of the simulation view and code. Implement Sci. 2013;8(1):117. (DOCX 992 kb) 7. Meyer A-M, Davis M, Mays GP. Defining organizational capacity for public health services and systems research. J Public Health Manag Pract. 2012; 18(6):535–44. Abbreviations 8. Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund ABM: Agent-based modeling; CDC: Centers for Disease Control and Q. 1966;44(3):166–206. Prevention; EBI: Evidence-based interventions; HIV: Human 9. Handler A, Issel M, Turnock B. Conceptual framework to measure performance Immunodeficiency Virus of the public health system. Am J Public Health. 2001;91(8):1235–9. McKay et al. Implementation Science (2018) 13:77 Page 10 of 10 10. Mays GP, Smith SA, Ingram RC, Racster LJ, Lamberth CD, Lovely ES. Public 35. Mishra S, Pickles M, Blanchard JF, Moses S, Boily M-C. Distinguishing sources of health delivery systems: evidence, uncertainty, and emerging research HIV transmission from the distribution of newly acquired HIV infections: why is needs. Am J Prev Med. 2009;36(3):256–65. it important for HIV prevention planning? Sex Transm Infect. 2014;90(1):19–25. 11. Scheirer MA, Dearing JW. An agenda for research on the sustainability of 36. Broz D, Wejnert C, Pham HT, DiNenno E, Heffelfinger JD, Cribbin M, et al. public health programs. Am J Public Health. 2011;101(11):2059–67. HIV infection and risk, prevention, and testing behaviors among injecting 12. Andria H, Katherine F, Carl T, Paul MW, Duncan C, Heather H. Immediate drug users—National HIV Behavioral Surveillance System, 20 US Cities, 2009. versus sustained effects: interrupted time series analysis of a tailored MMWR Surveill Summ. 2014;63(6):1–51. intervention. Implement Sci. 2013;8(1):130. 37. Sionean C, Le BC HK, Oster AM, Wejnert C, Hess KL, et al. HIV risk, prevention, and testing behaviors among heterosexuals at increased risk for HIV 13. Alexander JA, Hearld LR. Methods and metrics challenges of delivery-system infection—National HIV Behavioral Surveillance System, 21 US Cities, 2010. research. Implement Sci. 2012;7(1):15. Morb Mortal Wkly Rep Surveill Summ (Washington, DC: 2002). 2014;63(SS-14):1. 14. Greenhalgh T, Macfarlane F, Barton-Sweeney C, Woodard F. “If we build it, 38. Gilbert N. Agent-based models. London: Sage; 2008. will it stay?” a case study of the sustainability of whole-system change in 39. Hammond RA. Considerations and best practices in agent-based modeling London. Milbank Q. 2012;90(3):516–47. to inform policy. 2015. 15. Irene I, Kate G, Sue P, Sabrina E, Andrew B. Exploring scale-up, spread, and 40. Team RC. R: a language and environment for statistical computing R sustainability: an instrumental case study tracing an innovation to enhance foundation for statistical computing; 2017 Available from: https://www.r- dysphagia care. Implement Sci. 2013;8(1):128. project.org/ 16. Epstein JM. Generative social science: studies in agent-based computational 41. Dolcini MM, Gandelman AA, Vogan SA, Kong C, Leak TN, King AJ, et al. modeling. Princeton: Princeton University press; 2006. Translating HIV interventions into practice: community-based organizations’ 17. Ip EH, Rahmandad H, Shoham DA, Hammond R, Huang TT-K, Wang Y, et al. experiences with the diffusion of effective behavioral interventions (DEBIs). Reconciling statistical and systems science approaches to public health. Soc Sci Med. 2010;71(10):1839–46. Health Educ Behav. 2013;40(1 suppl):123S–31S. 42. Garner BR, Hunter BD, Modisette KC, Ihnes PC, Godley SH. Treatment staff 18. Dunn AG, Gallego B. Diffusion of competing innovations: The effects of turnover in organizations implementing evidence-based practices: turnover network structure on the provision of healthcare. J Artif Soc Soc Simul. rates and their association with client outcomes. J Subst Abus Treat. 2012; 2010;13(4):8. 42(2):134–42. 19. Zhang D, Giabbanelli PJ, Arah OA, Zimmerman FJ. Impact of different 43. Centers for Disease Control and Prevention (CDC). High-impact HIV policies on unhealthy dietary behaviors in an urban adult population: an prevention: CDC’s approach to reducing HIV infections in the United States. agent-based simulation model. Am J Public Health. 2014;104(7):1217–22. 2011. https://www.cdc.gov/hiv/policies/hip/hip.html. Accessed 08/01/17. 20. Orr MG, Galea S, Riddle M, Kaplan GA. Reducing racial disparities in obesity: 44. Aral SO, Blanchard J. Biomedical and social dimensions of HIV prevention. simulating the effects of improved education and social network influence Am J Public Health. 2012;102(9):e9-e. on diet behavior. Ann Epidemiol. 2014;24(8):563–9. 45. Kippax S, Stephenson N. Beyond the distinction between biomedical and 21. Luke DA, Hammond RA, Combs T, Sorg A, Kasman M, Mack-Crane A, et al. social dimensions of HIV: prevention through the lens of a social public Tobacco town: computational modeling of policy options to reduce health. Am J Public Health. 2012;102(5):789–99. tobacco retailer density. Am J Public Health. 2017;107(5):740–6. 46. Marrazzo JM, del Rio C, Holtgrave DR, Cohen MS, Kalichman SC, Mayer KH, 22. Fioretti G. Agent-based simulation models in organization science. Organ et al. HIV prevention in clinical care settings: 2014 recommendations of the Res Methods. 2013;16(2):227–42. international antiviral society—USA panel. JAMA. 2014;312(4):390–409. 23. Brookmeyer R, Boren D, Baral SD, Bekker L-G, Phaswana-Mafuya N, Beyrer C, 47. Hoffer L. Unreal models of real behavior: the agent-based modeling et al. Combination HIV prevention among MSM in South Africa: results from experience. Pract Anthropol. 2013;35(1):19–23. agent-based modeling. PLoS One. 2014;9(11):e112668. 24. Brown K, Furneaux C, Gudmundsson A. Infrastructure transitions towards sustainability: a complex adaptive systems perspective. Int J Sustain Dev. 2012;15(1):54–71. 25. Kamb ML, Dillon BA, Fishbein M, Willis KL. Quality assurance of HIV prevention counseling in a multi-center randomized controlled trial. Project RESPECT Study Group. Public Health Rep. 1996;111(Suppl 1):99–107. 26. Kamb ML, Fishbein M, Douglas JM, Rhodes F, Rogers J, Bolan G, et al. Efficacy of risk-reduction counseling to prevent human immunodeficiency virus and sexually transmitted diseases: a randomized controlled trial. Project RESPECT Study Group. JAMA. 1998;280(13):1161–7. 27. Johns DM, Bayer R, Fairchild AL. Evidence and the politics of deimplementation: the rise and decline of the “counseling and testing” paradigm for HIV prevention at the US Centers for Disease Control and Prevention. Milbank Q. 2016;94(1):126–62. 28. Danya. RESPECT. Danya International, Inc; 2012. http://www.effectiveinterventions. org/en/HighImpactPrevention/Interventions/RESPECT.aspx. Accessed 08/01/2017. 29. Catania JA, Dolcini MM, Gandelman AA, Narayanan V, Mckay VR. Fiscal loss and program fidelity: impact of the economic downturn on HIV/STI prevention program fidelity. Transl Behav Med. 2014;4(1):34–45. 30. Dolcini MM, Catania JA, Gandelman A, Ozer EM. Implementing a brief evidence-based HIV intervention: a mixed methods examination of compliance fidelity. Transl Behav Med. 2014;4(4):424–33. 31. McKay VR, Dolcini MM, Catania JA. Impact of human resources on implementing an evidence-based HIV prevention intervention. AIDS Behav. 2017;21(5):1394–406. 32. McKay VR, Margaret Dolcini M, Hoffer LD. The dynamics of de-adoption: a case study of policy change, de-adoption, and replacement of an evidence- based HIV intervention. Transl Behav Med. 2017;7(4):821–31. 33. Wilensky U. NetLogo user manual version 5.0. 5. Center for Connected Learning and Computer-Based Modeling Northwestern University, Evanston. 2014. 34. Centers for Disease Control and Prevention (CDC) HIV survellance reports. 2016. http://www.cdc.gov/hiv/library/reports/hiv-surveillance.html. Accessed 08/01/17.
Implementation Science – Springer Journals
Published: Jun 5, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera