Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’

Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good... Abstract Assessing societal impacts of research is more difficult than assessing advances in knowledge. Methods to evaluate research impact on policy processes and outcomes are especially underdeveloped, and are needed to optimize the influence of research on policy for addressing complex issues such as chronic diseases. Contribution analysis (CA), a theory-based approach to evaluation, holds promise under these conditions of complexity. Yet applications of CA for this purpose are limited, and methods are needed to strengthen contribution claims and ensure CA is practical to implement. This article reports the experience of a public health research center in Canada that applied CA to evaluate the impacts of its research on policy changes. The main goal was to experiment with methods that were relevant to CA objectives, sufficiently rigorous for making credible claims, and feasible. Methods were ‘good enough’ if they achieved all three attributes. Three cases on government policy in tobacco control were examined: creation of smoke-free multiunit dwellings, creation of smoke-free outdoor spaces, and regulation of flavored tobacco products. Getting to ‘good enough’ required careful selection of nested theories of change; strategic use of social science theories, as well as quantitative and qualitative data from diverse sources; and complementary methods to assemble and analyze evidence for testing the nested theories of change. Some methods reinforced existing good practice standards for CA, and others were adaptations or extensions of them. Our experience may inform efforts to influence policy with research, evaluate research impacts on policy using CA, and apply CA more broadly. 1. Introduction Interest in research evaluation is increasing for many reasons, to inform action, learning, accountability, and funding allocation decisions (Guthrie 2013; Penfield et al. 2014). To achieve these different purposes, there are diverse efforts to examine and document research impact (Penfield et al. 2014). Across these efforts, there is most consistency on measuring academic impacts, intellectual contributions made by research to a particular field of study (Greenhalgh et al. 2016). Measuring ‘societal’ impacts is less consistent and more challenging (Penfield et al. 2014; Greenhalgh et al. 2016). The challenge is identifying causal pathways that clearly link research to health, social, or economic changes (Hanney et al. 2000). Identifying and understanding these pathways are especially difficult for applied research efforts related to public policymaking, for which the links between research and impact are influenced by a wide array of factors and players (Milat et al. 2015; Greenhalgh et al. 2016). The importance of overcoming this challenge was reinforced by a modified Delphi study completed by members of the author team (Willis et al. 2017). We explored indicators that may be important for evaluating practical impacts of public health research centers, and identified promising directions for development of new methods for assessing these indicators. Consistent with recommendations by others (Boaz et al. 2009; Milat et al. 2015), panelists generally agreed that a critical focus for public health research centers is to influence policy development, implementation, and outcomes, and that current methods for capturing the influence of research on policy are underdeveloped. Theory-based evaluations are proposed as particularly promising, with considerable interest in contribution analysis (CA) (Dauphinee 2015; Buckley 2016). 1.1 Using CA for assessing research impact on policy Most scientific inquiry focuses on questions of attribution. Yet attribution analysis tends to focus on direct, verifiable causality that is not aligned with a generally accepted view of the policy process involving complex interconnections among activities and observed outcomes (Patton, 2012). CA, in contrast, can credibly assess cause and effect relationships in circumstances when impacts result from a complex interplay of actions (including research) by multiple players, and a variety of contextual factors (Mayne 2008). Application of CA involves six steps, starting with identifying the attribution problem to be addressed (Step 1), developing a theory of change (TOC) (Step 2), gathering evidence on the TOC (Step 3), assembling evidence and assessing the contribution story (Step 4) through an iterative process of seeking additional evidence (Step 5), and revising and strengthening the contribution story (Step 6) (Mayne 2008, 2012, 2015). CA is generally viewed as conceptually appealing, and yet methods for implementing the six steps to optimize the credibility and utility of CA are not well developed (Dybdal et al. 2011; Lemire et al. 2012; Wimbush et al. 2012; Dauphinee 2015). As a result, several authors point to methodological elaborations that may strengthen applications of CA and theory-based evaluations more generally (Delahais and Toulemonde 2012; Kok and Schuit 2012; Leeuw 2012; Lemire et al. 2012; Sridharan and Nakaima 2012; Willis et al. 2017; Morton 2015). Proposed elaborations address several questions the evaluation field is grappling with, such as: How do we go about strengthening theories of change that incorporate concepts of reach? (Montague 2015) How can we build in varied assumptions about pathways of impact, especially how these are influenced over time? (Mayne 2015) How might we bound analyses (e.g. focus on nested contribution stories) so they are both rigorous and feasible? (Mayne 2015) How can we strengthen the use of scientific theories in CA and other theory-based evaluations? (Willis et al. 2017) How can we account for competing explanations and influencing factors? (Dybdal et al. 2011) 1.2 Study purpose and significance As part of an organizational-level evaluation, our team applied CA to evaluate the impacts of research on a specific domain of public health policy—government regulations for tobacco control in Canada, spanning municipal, provincial, and federal levels. We used three cases (described below) from the Propel Centre for Population Health Impact (Propel), a national research program of the Canadian Cancer Society, and hosted by the Faculty of Applied Health Sciences at the University of Waterloo, Ontario, Canada. Propel exists to prevent cancer and other chronic diseases and their shared behavioral and environmental causes, and achieves its mandate by catalyzing and conducting relevant studies using the most appropriate and rigorous methods, and moving evidence into action. Research and knowledge exchange activities completed by Propel are intended to contribute to improvements in policies and programs that impact health at a population level. Tobacco control policies are an important case in point, and have been a focus within Propel for 25 years. Propel activities are part of an increasingly diverse and complex array of actions by multiple sectors and players, interacting with a similarly diverse set of contextual factors, all influencing policy development, implementation, and outcomes. These conditions are consistent with circumstances under which CA is considered particularly useful. The primary purpose of this article is to advance CA methods. Our goal was to use methods that would be relevant to CA objectives, sufficiently rigorous to make defensible contribution claims, and also practical to implement as part of an organizational approach to evaluation (Newcomer and Brass 2016). Achieving these attributes is how we defined a ‘good enough’ CA. This article may be of interest to those concerned with how research can be used effectively to influence policy processes and outcomes. It may also be of interest to those wanting to evaluate research impacts on policy change, including funders, evaluators, and knowledge users. A third audience is those who are interested in applying CA, especially for evaluating research impacts on policy change, and especially in the domain of population and public health. The experience reported in these specific domains, however, may inform other uses of CA. 2. Application of CA We applied CA to evaluate the influence of Propel initiatives on the development and uptake of three areas of tobacco control policy in Canada: creation of smoke-free multiunit dwellings (MUDs), creation of smoke-free outdoor spaces (SFOS), and regulation of flavored tobacco products (FT). Following a description of the evaluation team, we describe our application of CA to the three cases, and use one case (MUDs) as an illustrative example. 2.1 CA team and approach We used a team approach, with members from inside and outside Propel. Internal team members included faculty (B. R., J. Y., C. W.) and research staff (A. K., L. S.), all of whom have in-depth knowledge of Propel’s mandate, activities, and way of working, yet who were not directly involved in the research and knowledge exchange activities for the three tobacco control policy case examples. Propel internal team members brought content expertise in tobacco control and skills in evaluation and project management. They also had access to people who participated in each of the three cases, mainly through other Propel staff and their collaborators. The external team member (S. M.) is a consultant with extensive expertise in evaluation, including CA. Team roles included an overall scientific lead for the study (B. R.), a lead for each case example (A. K. for two cases and S. M. for one), CA methodological guidance and experience (S. M., J. Y.), and linkages to other center evaluation activities (B. R., L. S., C. W.). We conducted these studies over a 1-year period. Estimated time commitments from members of the CA team were: 0.5 full-time equivalents (FTEs) for in-house Propel staff, 0.2 FTE for in-house faculty, and 0.15 FTE for the external consultant. Our team complied with the recommended iterative approach when applying the CA steps (Mayne 2008). For all steps, we held regular team meetings to discuss what was working well and any existing issues or obstacles. Problem-solving occurred within and across the three cases. 2.2 Select cases for the study (CA Step 1) CA’s first step is to set out the attribution problem to be addressed. In our application, this happened in two stages. The first was selection of cases that would allow us to address the overall study purpose of examining how Propel initiatives influenced the adoption of policy, and the second was identifying the nested theories of change that we would focus on within each case. The latter was done after partial implementation of CA Step 2, and is described in that section. Our goal was to select a small number of cases (2 or 3), and each needed to meet three criteria. First, evidence of policy adoption was needed, in at least one jurisdiction and preferably more. Second, sufficient information related to Propel contributions and other influencing factors needed to exist or could be collected. This was done by reviewing organizational records (a central tracking system) for relevant documents, and consulting with Propel personnel directly involved in each case to explore access to collaborators outside of Propel. Third, circumstances and events related to the case could be expected to apply to other policy domains and other jurisdictions. These assessments were done by team members, all with relevant expertise and experience. By applying these criteria, three cases were identified. MUDs are a set of tobacco control policies adopted by regional government agencies, individual landlords, and related policymakers to regulate smoking in dwellings with multiple living units (e.g. apartments). SFOS is a set of tobacco control policies at municipal and provincial levels designed to protect people in a variety of outdoor spaces (e.g. parks, playgrounds, patios) from exposure to second-hand smoke, and also to minimize the social modeling of smoking, particularly among children and youth. FTproduct regulations are under provincial and federal jurisdictions, and emerged as an important policy domain in 2010, especially to protect youth from initiation and use of new tobacco products, including menthol. 2.3 Develop an initial TOC for each case (CA Step 2) A TOC represents hypothesized cause and effect relationships and related assumptions within an overall pathway of change. Theories of change can be shown at various levels of detail, with general agreement among CA scholars to begin with a high-level TOC, before progressively adding detail as the model develops. We applied the principle of taking a progressive and iterative approach to developing the TOCs that would become the primary focus of our CA. We undertook four main tasks that are described below. First, we developed an overall logic model for each case. We used a common format across cases (consistent with Mayne, 2015) based on the Bennett hierarchy with seven interdependent components (Bennett and Rockwell 1995). Guided by Pasanen and Shaxson (2016), and as recommended for complex processes (Nutley et al. 2007; Boaz et al. 2009), we also included proximal markers of research influence on policy change (e.g. research products, knowledge exchange activities). The initial logic models were completed by the CA team in consultation with Propel personnel directly involved in each case (minimally, the lead faculty and staff members). Propel personnel were asked to identify documents that provided an overview of Propel activities, participating individuals and organizations, and policy-related outcomes; participated in at least one in-depth conversation with the CA team; and provided feedback on the logic model. Figure 1 shows the product of this first task for the MUDs case. Figure 1. View largeDownload slide Overall logic model for the creation of smoke-free MUDs. Figure 1. View largeDownload slide Overall logic model for the creation of smoke-free MUDs. As shown in Fig. 1, creating smoke-free MUDs had three main areas of development: early pilot work in the Waterloo region to support policy development (Section A); Propel’s research to increase understanding of policy implementation (Section B); and knowledge exchange and policy change processes (Section C) within Waterloo and dissemination to other communities. These three main developments were underpinned by shared inputs, including financial and personnel investments from Propel, a collaborative approach to research that actively engaged knowledge users, strong existing relationships between Propel researchers and local public health colleagues, and synthesis of research information in meaningful ways for knowledge users. Moving from left to right, the logic model includes a temporal dimension that shows the progression of activities and outcomes over time. Logic models for the other two cases were similar, with each including feedback loops showing how early products, results, and learnings informed subsequent work. A difference in the logic models across cases was the level of complexity. The FT case was less complex than the other cases, mostly due to a more limited set of stakeholders involved. FT policy changes were at federal and provincial levels, and provincial advocates (as direct links to policymakers) were the main stakeholders involved in the uptake of Propel products. In the other two cases, the policy process involved more diverse groups of stakeholders, and impact spread across more settings and contexts. The SFOS case, for example, involved policy change in several municipalities initially, followed by changes in provincial policy. As recommended by others (Delahis and Toulemonde 2012; Sridharan and Nakaima 2012; Mayne 2015), our second task was to choose a nested impact pathway within each comprehensive logic model. The nested pathway would become the focus of subsequent steps in the CA. Nested pathways were selected that would be feasible to complete, and that represented previously underexplored links in the research impact pathway with high relevance to Propel strategy. Nested pathways, therefore, would contribute to new knowledge, and provide useful information to guide future efforts by Propel and others to influence policy with research. Across all three cases, the pathway selected for further analysis focused on Propel’s role in knowledge translation and dissemination, and the influence of these activities on the engagement of key audiences, changes in capacity, and the adoption of relevant smoke-free policies. For the MUDs case, the nested impact pathway is shown Fig. 1. The third task within CA Step 2 was to synthesize aspects of relevant social science theories that could inform subsequent CA steps. This task was intended to address a common criticism of limited explicit use of scientific theories in CA and evaluation generally (Patton 2012; Willis et al. 2017). Our goal was to engage with scientific theories in ways that added value and credibility to our studies, while also ensuring feasibility for our own and other CA studies. Aiming to strike this balance, and acknowledging a very large and diverse set of potentially relevant theories (cf. Stachowiak 2013), each CA team member nominated a small subset of theories they considered to be most relevant to the nested impact pathways. We specifically identified theories that focus on what helps and hinders research use in policy, and the policy change process. Using nominated sources, we developed a theory-based checklist with three sections. The first section refers to general parameters for the influence of research, including various environments, timeframes, intentions, and levels within which research influence may occur. The second section refers to mechanisms of research influence (e.g. priming, persuasion, elaboration, etc.) and is primarily informed by Mark and Henry (2004), Pasanen and Shaxson (2016), and Pawson (2013). The third section of the checklist refers to contextual factors, all of which are known to influence research utilization, and include characteristics and capacities of people, their relationships and the wider environment. The contextual factors are mostly informed by Pawson’s (2013) four I’s of contextual influence (i.e. individuals, interpersonal relations, institutional settings, and infrastructure), Kingdon’s (2003) advocacy and policy change contexts, and Rogers’ (1995) characteristics of innovations that support diffusion. The fourth and final task in CA Step 2 was to embellish the nested impact pathways into theories of change, by naming a subset of assumptions and contextual factors that may be most influential in explaining the links in the impact pathway for each case. These theories of change were the focus of subsequent CA steps. The TOC for the MUDs nested impact pathway is shown in Fig. 2. Figure 2. View largeDownload slide TOC for the MUDs nested impact pathway (adapted from Mayne, 2015). Figure 2. View largeDownload slide TOC for the MUDs nested impact pathway (adapted from Mayne, 2015). 2.4 Gather evidence on the TOCs (CA Step 3) The primary purpose of CA Step 3 is to test the validity of the TOC (observed results, assumptions, influencing factors). Our goal was to gather the most relevant evidence, all of sufficient quality and from diverse sources to test each TOC. Relevance and sufficient quality were minimum criteria for all sources of evidence. Seeking diverse sources of evidence respected the inherent complexity of our cases (e.g. many actors and agencies involved in the policy development process over time) (Rogers 2008), and also responded to a criticism that most studies rely on interviews with principal investigators and/or peer review to assess research impacts (Biggs et al 2014; Penfield et al 2014; Milat et al 2015). One way diversity was achieved was using mixed methods (Mertens and Hesse-Biber 2013), allowing us to explore the extent to which quantitative (numerical) and qualitative (text) information was consistent and/or complementary for different parts of the TOC. Diversity was also achieved by interviewing people who had different roles in the policy process (e.g. researchers, knowledge users, knowledgeable commentators in relevant policy domains), and thereby different vantage points on the context, process, and outcomes for each TOC. Perspectives external to Propel were given particular attention, as sources of information that broadened and increased the robustness of the evidence to either support contribution assertions or provide alternative explanations. A practical aim was to gather evidence as efficiently as possible. To this end, we sought to identify existing information (collected for a purpose other than the CA) that met our criteria, and also collected new information to fill gaps and expand on existing evidence. Sources of evidence were identified progressively. For each case, we started data collection by interviewing up to three personnel from Propel who were most directly involved in each of the three cases. In addition to collecting primary data from these personnel (interview strategy described in more detail below), we asked these informants to identify other sources of information, including additional informants (i.e. snowball sampling; cf. Corbin and Strauss 2008), documents, Web-based statistics, media, etc. When seeking nominations for additional sources of information, we emphasized the importance of diverse types of evidence and diverse perspectives, as described above. To illustrate the mix of evidence, Table 1 shows the sources of evidence for the MUDs case. Table 1. Sources of evidence for the MUDs case Data sources  Description  Reports and videos  Twenty-six documented examples of smoke-free MUDs activities and initiatives, e.g. Waterloo Region Smoke-Free Housing Policy Evaluation report  Interviews  Consultations with 12 informants, e.g. advocates, researchers, and public health staff from multiple jurisdictions  Tracking forms  Detailed documentation of over 60 consultations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions, and the status of subsequent smoke-free policy adoption over time (from 2009 to 2015)  Web statistics  Downloads and shares (variable for this time period)  Citation analysis  Some citation evidence from public health reports  Data sources  Description  Reports and videos  Twenty-six documented examples of smoke-free MUDs activities and initiatives, e.g. Waterloo Region Smoke-Free Housing Policy Evaluation report  Interviews  Consultations with 12 informants, e.g. advocates, researchers, and public health staff from multiple jurisdictions  Tracking forms  Detailed documentation of over 60 consultations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions, and the status of subsequent smoke-free policy adoption over time (from 2009 to 2015)  Web statistics  Downloads and shares (variable for this time period)  Citation analysis  Some citation evidence from public health reports  Each of the three cases differed in the type and amount of existing information available. Evidence that was common across cases included websites, attendance lists, citations, media coverage, and social media analytics. Some unique sources were also used, such as records kept by Region of Waterloo of presentations and consultations undertaken with municipalities and other stakeholders interested in smoke-free MUDs, and a publicly available inventory of SFOS policies across Canada. The amount of data from different sources also varied across cases, with the most comprehensive documentation for media coverage on FT. Common limitations to existing evidence were incomplete information on the reach and uptake of Propel documents, outcomes from partner meetings, audiences for presentations, and social media statistics. Some of these data were unavailable, inconsistently recorded, or untraceable (e.g. Propel documents or videos were often downloaded from links on partner’s sites and those statistics were not reflected in the Propel data). Across all cases, informants internal and external to Propel were interviewed, with variable numbers and diversity of informants. In the FT case, informants were fewer and came from less diverse settings compared to those from the MUDs and SFOS cases. Most FT case informants were researchers and tobacco control advocates at provincial and national levels, many of whom are part of an existing pan-Canadian network for implementing a national survey and using data from this survey to inform advocacy. In contrast, a larger number and more diverse set of informants were interviewed for the MUDs and SFOS cases. Main informants included researchers from within and beyond Propel, local public health practitioners from Waterloo and other communities, and nongovernment organizations and tobacco control advocates at provincial and national levels. Similar to the FT case, some of the informants were members of an existing network. Several informants were consistent across the MUDs and SFOS cases, so when possible and appropriate, questions on both cases were completed in a single interview. Interviews for each case were terminated when all nominated informants had been contacted. We used semi-structured interviews (Corbin and Strauss 2008) that were informed by the TOCs and the theory-based checklist. Interview questions and probes were customized for each case and allowed informants to tell their story of the events leading up to policy change, including factors that were particularly relevant in their context. In the MUDs case, for example, questions contextualized the influence of Propel’s work in relation to current activities in Canada focused on smoke-free policies in MUDs, the contributions of other groups, and what policy change may have occurred in the absence of Propel’s work. Processes of influence were also explored, such as the degree to which Propel’s work may have achieved influence through priming (e.g. increasing the salience of a concept, such as everyone has the right to breathe clean air), or through capacity building by improving understanding of issues related to MUDs. Informants were generally a rich source of information, and they provided consistent feedback on Propel’s contributions to policy change. As expected, recall was better when research and policy change occurred more recently (such as FT policy change), and from informants directly involved in the events. 2.5 Assemble, assess, and strengthen the contribution stories (CA Steps 4, 5, and 6) CA Steps 4 through 6 were performed iteratively, beginning with assembling the evidence and assessing its strengths and limitations (CA Step 4), and then completing cycles of gathering additional evidence (CA Step 5) and revising and strengthening the contribution story (CA Step 6). We combine the three steps in this section, since additional data gathering (Step 5) and techniques for assessing strength of the contribution stories (Steps 4 and 6) were repeated in the iterations. We describe how we assembled the contribution stories, assessed their strengths and limitations, and applied Steps 4 through 6 iteratively. 2.5.1 Assembling the contribution stories We adapted approaches used by others (Montague and Valentim 2010; Morton 2015) for assembling evidence. Our goals were to optimize accuracy and sufficiency of data assembly as a basis for subsequent assessments, while also making sure displays were practical to complete. A first display was intended to facilitate a rigorous review of the strengths and limits of the emerging contribution stories. Building on Morton’s ‘evidence of impact’ table (Morton 2015: 411), evidence was assembled for the TOC components: activities in the nested impact pathway, assumptions linking these, and contextual influences. Limitations to the evidence were also identified (see Table 2). Table 2. Illustrative evidence for the smoke-free MUDs TOC Impact pathway  Illustrative evidence   Limitations of the evidence  Activities  Context    Advocacy intermediaries and advocates are engaged  Reports and videos—Documentation of smoke-free MUDs policy change in Waterloo was distributed through channels, e.g. provincial knowledge exchange forum Interviews—Informant assertions of the uptake of Propel messaging. One advocate said, ‘Public health pays a lot of attention to what comes out of Propel … products are consumable by the government and public.’ Tracking form—Documentation of 60+ presentations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions  Interviews—Informant assertions of several factors that influenced engagement, e.g. Propel’s consultative and participative approach, the accessibility and credibility of the products, and the tangibility of the evidence  Detailed recall was not consistent across all informants, as some people could not remember who was involved in what    Decision/policymakers in other communities receive information on second-hand smoke policies in Waterloo MUDs and engage in briefings  Interviews—Informant assertions, e.g. ‘Everybody in MUDs is using our process and our documentation as a template.’ (public health staff)  Interviews—Informant assertions that coordinated messaging was influential in getting information to decision-makers  Assumptions about the conditions needed to engage decision/policymakers are based on the recall of a selective set of participants    Reach assumptions: Information and products are accessible  Reports and videos and Web statistics—A range of stakeholders were exposed to the information through multiple channels and mediums Interviews—Informants discussed the influence of Propel’s knowledge translation stating, ‘Accessible information products are important, It is not enough for science to be published in a journal and wait for the attention to follow.’ (advocate)  Web statistics do not show a comprehensive picture of reach as reports and videos uploaded from partner sites were not included in Propel’s data    Constructive reactions  Interviews—Numerous informants noted the positive reaction of groups to Propel scientists, staff, and presentations  Interviews—Informants referred to an environment that promoted constructive reactions, e.g. ‘People are more likely to be persuaded if they have a face or a name.’ (advocate)  Retroactively, a survey was not done, as a robust source of evidence    Capacity change assumptions: Rationale and need for MUDs smoking ban is understood Smoking ban is perceived as achievable  Reports and videos—Propel’s research showed the impact of smoke-free MUDs policy on tenant perceptions, attitudes, and self-assessed smoking behavior Interviews—Informants asserted that this research was ‘useful to them to be able to make an informed decision on where they wanted to go.’ (researcher) Informants also discussed the influence of Waterloo’s work on other jurisdictions, e.g. ‘The fact that the change has been demonstrated by someone first was viewed as important in building confidence about the do-ability of MUDs.’ (former community of practice staff)  There was insufficient research evidence to determine the economic impacts of the MUDs policy, e.g. on regional workload and costs    Knowledge gain and commitment to reducing second-hand smoke in MUDs through smoking bans  Interviews—Informants referred to the Waterloo experience as a motivator for change saying, ‘If it had not happened in Waterloo, it would not have happened in [name of community].’ (researcher)  Interviews—Informant assertions that successful policy change by the Region of Waterloo built commitment  Without a comparison group, it is not possible to be unequivocal about the influence of Propel    Behavior change assumptions: Waterloo policies are appropriate or can be adapted in other communities Second-hand smoke is a priority for MUDs in other communities Smoking ban is supported by the majority of MUDs residents  Interviews—Informants discussed the positive influence of the policy change in Waterloo Region in paving the way for other jurisdictions, e.g. ‘Waterloo being one of the first in Ontario was obviously a big role model for [our community] to emulate.’ (public health staff)  There is incomplete evidence on the priorities of housing authorities and the level of support for similar policies in other jurisdictions across the province at that time    Smoke-free MUDs policy adoption  Interviews – Informant assertions that Propel work was influential in policy change in other jurisdictions. ‘Propel made it more reasonable, possible for our community.’ (public health unit staff) Tracking form—Direct link between timing of consultations and subsequent policy change in seven Ontario communities  Interviews—Informant assertions that the linked interests, through a regional government committee structure, may have positively influenced smoke-free policy adoption in MUDs  Reflections on other necessary conditions supporting policy adoption in other communities are not available  Impact pathway  Illustrative evidence   Limitations of the evidence  Activities  Context    Advocacy intermediaries and advocates are engaged  Reports and videos—Documentation of smoke-free MUDs policy change in Waterloo was distributed through channels, e.g. provincial knowledge exchange forum Interviews—Informant assertions of the uptake of Propel messaging. One advocate said, ‘Public health pays a lot of attention to what comes out of Propel … products are consumable by the government and public.’ Tracking form—Documentation of 60+ presentations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions  Interviews—Informant assertions of several factors that influenced engagement, e.g. Propel’s consultative and participative approach, the accessibility and credibility of the products, and the tangibility of the evidence  Detailed recall was not consistent across all informants, as some people could not remember who was involved in what    Decision/policymakers in other communities receive information on second-hand smoke policies in Waterloo MUDs and engage in briefings  Interviews—Informant assertions, e.g. ‘Everybody in MUDs is using our process and our documentation as a template.’ (public health staff)  Interviews—Informant assertions that coordinated messaging was influential in getting information to decision-makers  Assumptions about the conditions needed to engage decision/policymakers are based on the recall of a selective set of participants    Reach assumptions: Information and products are accessible  Reports and videos and Web statistics—A range of stakeholders were exposed to the information through multiple channels and mediums Interviews—Informants discussed the influence of Propel’s knowledge translation stating, ‘Accessible information products are important, It is not enough for science to be published in a journal and wait for the attention to follow.’ (advocate)  Web statistics do not show a comprehensive picture of reach as reports and videos uploaded from partner sites were not included in Propel’s data    Constructive reactions  Interviews—Numerous informants noted the positive reaction of groups to Propel scientists, staff, and presentations  Interviews—Informants referred to an environment that promoted constructive reactions, e.g. ‘People are more likely to be persuaded if they have a face or a name.’ (advocate)  Retroactively, a survey was not done, as a robust source of evidence    Capacity change assumptions: Rationale and need for MUDs smoking ban is understood Smoking ban is perceived as achievable  Reports and videos—Propel’s research showed the impact of smoke-free MUDs policy on tenant perceptions, attitudes, and self-assessed smoking behavior Interviews—Informants asserted that this research was ‘useful to them to be able to make an informed decision on where they wanted to go.’ (researcher) Informants also discussed the influence of Waterloo’s work on other jurisdictions, e.g. ‘The fact that the change has been demonstrated by someone first was viewed as important in building confidence about the do-ability of MUDs.’ (former community of practice staff)  There was insufficient research evidence to determine the economic impacts of the MUDs policy, e.g. on regional workload and costs    Knowledge gain and commitment to reducing second-hand smoke in MUDs through smoking bans  Interviews—Informants referred to the Waterloo experience as a motivator for change saying, ‘If it had not happened in Waterloo, it would not have happened in [name of community].’ (researcher)  Interviews—Informant assertions that successful policy change by the Region of Waterloo built commitment  Without a comparison group, it is not possible to be unequivocal about the influence of Propel    Behavior change assumptions: Waterloo policies are appropriate or can be adapted in other communities Second-hand smoke is a priority for MUDs in other communities Smoking ban is supported by the majority of MUDs residents  Interviews—Informants discussed the positive influence of the policy change in Waterloo Region in paving the way for other jurisdictions, e.g. ‘Waterloo being one of the first in Ontario was obviously a big role model for [our community] to emulate.’ (public health staff)  There is incomplete evidence on the priorities of housing authorities and the level of support for similar policies in other jurisdictions across the province at that time    Smoke-free MUDs policy adoption  Interviews – Informant assertions that Propel work was influential in policy change in other jurisdictions. ‘Propel made it more reasonable, possible for our community.’ (public health unit staff) Tracking form—Direct link between timing of consultations and subsequent policy change in seven Ontario communities  Interviews—Informant assertions that the linked interests, through a regional government committee structure, may have positively influenced smoke-free policy adoption in MUDs  Reflections on other necessary conditions supporting policy adoption in other communities are not available  2.5.2 Assessing strengths and limitations of the contribution stories The credibility of contribution claims is paramount in CA, and methods to enhance credibility are most underdeveloped. We examined strengths and limitations of the contribution stories using different approaches, with three of the most promising described below. One assessment examined the consistency of data in the evidence of impact tables (Table 2). Consistency was considered highest if findings were similar across multiple sources; medium if findings from different sources were complementary and supporting similar directions of influence in the pathway; and lowest if findings from different sources were contradictory and/or significant gaps or limitations were apparent. Although the specific data and themes varied across cases, each case had evidence from multiple sources and examples of data with different degrees of consistency. This common pattern is shown in Table 2, with the illustrative evidence for the first activity in the MUDs impact pathway. This activity (engagement of advocacy intermediaries and advocates) was supported with evidence from reports and videos, interviews, and the tracking form. Interview data highlighted the importance of characteristics of the evidence (tangibility), Propel’s approach (consultative and participatory), and the products themselves (accessible and credible) that may have influenced the success of the engagement. Since at least 7 years had passed since the early engagement for the MUDs research, detailed and nuanced information (e.g. about the roles and contributions from Propel, and influential contextual factors) for this stage of engagement was limited. A second assessment of strengths and limits was adapted from Funnell and Rogers (2011), and is shown in Table 3. We examined congruence with the program theory, comparisons to what might have happened in the absence of the intervention, and critical review of exceptions to anticipated patterns and other possible explanations. As shown with illustrative evidence in Table 3, evidence for making contribution claims is strong in the MUDs case. The congruence between the TOC and available evidence is high. In the counterfactual comparison, the evidence points to an increased trajectory of smoke-free MUDs across Ontario following Propel’s work to evaluate and disseminate the findings of smoke-free MUDs in Waterloo. In addition, evidence from interviews suggests that informants estimate that the spread of smoke-free MUDs would have been significantly slower without the work of Propel. With respect to critical review, the possible influence of US-based research on second-hand smoke in buildings was probed with knowledge user informants. They noted that local evidence was considered more relevant and had a stronger influence on their actions than research from outside of Canada. The timing of information release to policy change corroborates that assertion. Table 3. Assessing strength of Propel contributions to smoke-free MUDs using congruence, counterfactual comparison, and critical review (adapted from Funnell and Rogers, 2011) Methods and techniques  Illustrative evidence  Congruence   Comparing achievement of sequence of results outcomes: Where the intended outcomes have been achieved, the intermediate outcomes have also been achieved. Where the intended outcomes have not been achieved, the intermediate outcomes have also not been achieved  Tracking form: Region of Waterloo consultations and subsequent policy change in seven Ontario communities follow the expected pattern as detailed in Table 2. The necessary events and conditions needed to influence the policy change were present and aligned with policy change and innovation and diffusion theories   Disaggregating results for complicated interventions: Checking the results matches the TOC when a more complex causal package is important  Details are included in Figure 2 and Table 2   Modus operandi: Some interventions have distinct patterns of effect that can be used as evidence for causal influence  Interviews: Informants asserted that engagement was a key element influencing policy change, and noted Propel’s participation in high engagement activities, e.g. engagement of decision/policymakers was facilitated by the sharing of accessible documentation and process, and coordination across messaging groups Tracking form: Documentation of greater than 60 consultations and presentations by Region of Waterloo staff   Comparing timing of outcomes with program theory: Program theory may predict how long before final outcomes are evident and also whether these are likely to be maintained, increase, or decay over time  Tracking form: Region of Waterloo consultations occurred prior to policy change in seven Ontario communities Interviews: Informants from two communities asserted that engagement with the Waterloo experience directly influenced their adoption of a smoke-free MUDs policy   Comparing dose–response patterns with program theory: Program theory might predict whether increased exposure to an intervention is expected to have a positive, negative, or curvilinear relationship to the intended outcomes  Interviews: Informant assertions that one Ontario community consulted extensively with Waterloo. Notably, that community has the most widespread coverage of housing units included in their smoke-free MUDs policy change   Comparing expert predictions with actual results: For evaluations conducted over a period of time, it is possible to make predictions based on program theory or an emerging theory of wider contributors to outcomes, and then to follow up these predictions over time  N/A   Asking participants: While participants might sometimes have their own reasons for attributing or not attributing changes to an intervention, detailed accounts of their change trajectory can be credible  Interviews: All direct participants consulted in this case study asserted that the Waterloo initiative had influence. There were no participants who did not assert influence   Asking other informants  Interviews: Informants from related associations and exposed communities all asserted Waterloo/Propel influence on their pathway to policy change   Making comparisons across cases: The method of qualitative comparative analysis compares the configurations of different cases to identify the components that produce specific outcomes. Program theory can help to identify the variables that should be included in this analysis  Comparison across cases shows that Waterloo/Propel might be important but not sufficient. Further investigation may reveal that Propel/Waterloo may be necessary but not sufficient for MUDs in Ontario in the present time frame  Counterfactual comparison     Comparing the trajectory before and after the intervention: Time series data can provide a credible estimate of the counterfactual in fairly stable situations  Tracking form: There were no Ontario MUDs prior to the start of the Propel work; however, the Region of Waterloo tracking shows after the Propel work seven communities in Ontario adopted similar smoke-free policies for public housing. Each of these seven communities consulted Waterloo processes/experiences prior to adopting their policies   Thought experiments to develop plausible alternative scenarios: Evidence about policies and procedures and other opportunities in some cases can be used to develop a realistic scenario of the chain of events in the absence of an intervention  Interviews: Informants were asked to speculate what would have happened without the Waterloo/Propel work. No informant suggested that the development of MUDs would have gone the same way without it. Those who estimated suggested that ‘we might not be there yet’ and that the impact was likely extensive, i.e. several years  Critical review     Identifying alternative explanations and seeing if they can be ruled out: Alternative explanations might come from insiders (participants, informants), previous research, or speculation  Interviews: Informants noted that American research revealed an awareness of the issue of second-hand smoke risk in buildings, but that people typically prefer local examples   Identifying and explaining exceptions: Exceptional cases might be successes that were expected to be failures or vice versa. Ideally, an evaluation can explain these, or at least document that they exist, and not lose this information by focusing only on the overall pattern  Tracking form: Among the seven Ontario communities to adopt smoke-free MUDs policies after consultations with Waterloo, one community may be the exception to the rule. This community appears to have changed prior to consultation (albeit they still passed their policy 2 years after Waterloo’s passage)  Methods and techniques  Illustrative evidence  Congruence   Comparing achievement of sequence of results outcomes: Where the intended outcomes have been achieved, the intermediate outcomes have also been achieved. Where the intended outcomes have not been achieved, the intermediate outcomes have also not been achieved  Tracking form: Region of Waterloo consultations and subsequent policy change in seven Ontario communities follow the expected pattern as detailed in Table 2. The necessary events and conditions needed to influence the policy change were present and aligned with policy change and innovation and diffusion theories   Disaggregating results for complicated interventions: Checking the results matches the TOC when a more complex causal package is important  Details are included in Figure 2 and Table 2   Modus operandi: Some interventions have distinct patterns of effect that can be used as evidence for causal influence  Interviews: Informants asserted that engagement was a key element influencing policy change, and noted Propel’s participation in high engagement activities, e.g. engagement of decision/policymakers was facilitated by the sharing of accessible documentation and process, and coordination across messaging groups Tracking form: Documentation of greater than 60 consultations and presentations by Region of Waterloo staff   Comparing timing of outcomes with program theory: Program theory may predict how long before final outcomes are evident and also whether these are likely to be maintained, increase, or decay over time  Tracking form: Region of Waterloo consultations occurred prior to policy change in seven Ontario communities Interviews: Informants from two communities asserted that engagement with the Waterloo experience directly influenced their adoption of a smoke-free MUDs policy   Comparing dose–response patterns with program theory: Program theory might predict whether increased exposure to an intervention is expected to have a positive, negative, or curvilinear relationship to the intended outcomes  Interviews: Informant assertions that one Ontario community consulted extensively with Waterloo. Notably, that community has the most widespread coverage of housing units included in their smoke-free MUDs policy change   Comparing expert predictions with actual results: For evaluations conducted over a period of time, it is possible to make predictions based on program theory or an emerging theory of wider contributors to outcomes, and then to follow up these predictions over time  N/A   Asking participants: While participants might sometimes have their own reasons for attributing or not attributing changes to an intervention, detailed accounts of their change trajectory can be credible  Interviews: All direct participants consulted in this case study asserted that the Waterloo initiative had influence. There were no participants who did not assert influence   Asking other informants  Interviews: Informants from related associations and exposed communities all asserted Waterloo/Propel influence on their pathway to policy change   Making comparisons across cases: The method of qualitative comparative analysis compares the configurations of different cases to identify the components that produce specific outcomes. Program theory can help to identify the variables that should be included in this analysis  Comparison across cases shows that Waterloo/Propel might be important but not sufficient. Further investigation may reveal that Propel/Waterloo may be necessary but not sufficient for MUDs in Ontario in the present time frame  Counterfactual comparison     Comparing the trajectory before and after the intervention: Time series data can provide a credible estimate of the counterfactual in fairly stable situations  Tracking form: There were no Ontario MUDs prior to the start of the Propel work; however, the Region of Waterloo tracking shows after the Propel work seven communities in Ontario adopted similar smoke-free policies for public housing. Each of these seven communities consulted Waterloo processes/experiences prior to adopting their policies   Thought experiments to develop plausible alternative scenarios: Evidence about policies and procedures and other opportunities in some cases can be used to develop a realistic scenario of the chain of events in the absence of an intervention  Interviews: Informants were asked to speculate what would have happened without the Waterloo/Propel work. No informant suggested that the development of MUDs would have gone the same way without it. Those who estimated suggested that ‘we might not be there yet’ and that the impact was likely extensive, i.e. several years  Critical review     Identifying alternative explanations and seeing if they can be ruled out: Alternative explanations might come from insiders (participants, informants), previous research, or speculation  Interviews: Informants noted that American research revealed an awareness of the issue of second-hand smoke risk in buildings, but that people typically prefer local examples   Identifying and explaining exceptions: Exceptional cases might be successes that were expected to be failures or vice versa. Ideally, an evaluation can explain these, or at least document that they exist, and not lose this information by focusing only on the overall pattern  Tracking form: Among the seven Ontario communities to adopt smoke-free MUDs policies after consultations with Waterloo, one community may be the exception to the rule. This community appears to have changed prior to consultation (albeit they still passed their policy 2 years after Waterloo’s passage)  A third approach to assessing the robustness of the contribution stories was interpreting the stories in relation to pertinent scientific theories. In general, findings supported by well-established theories were considered most robust and plausible. We limited our assessment to the scientific theories that informed our theory-based checklist and examined their relevance to the contribution stories in more depth than the checklist. In the MUDs case, one example is how Kingdon’s (2003) three streams theory enhanced the explanatory power and understanding of the policy change process outside of Waterloo region. According to Kingdon (2003), moving an idea onto or higher up on the policy agenda involves three processes: problems, proposals, and politics. In the MUDs case, the evidence showed that Propel research highlighted air quality as a serious issue in MUDs (i.e. the ‘problem’). Propel research also showed the feasibility of making change, as survey results showed support from residents for smoking bans in MUDs (i.e. the ‘proposal’), and the political climate in this case was an important enabler of change (i.e. the ‘politics’). Theories of innovation diffusion could also help explain the patterns of influence. For example, by showing air quality could be measured, Propel may have assisted in making it more trialable, observable, and tangible, making the evidence for change more concrete. Another use of scientific theories in our analysis was to examine contextual influences. We mapped observed contextual influences onto Pawson’s (2013) four categories of contextual factors. In all cases, several factors within each of the four categories were readily mapped onto multiple steps along the impact pathway. Figure 3 shows a subset of factors in each category for the MUDs case. This mapping was useful to describe the range of contextual influences. It also raised questions for future study about the interplay of contextual factors within and across each of the categories. Figure 3. View largeDownload slide Map of illustrative contextual influences for the smoke-free MUDs nested impact pathway. Figure 3. View largeDownload slide Map of illustrative contextual influences for the smoke-free MUDs nested impact pathway. To arrive at an overall contribution story, we examined the consistency and complementarity of results from the above analyses, both within and across cases. We concluded that Propel’s work to synthesize, report, and socialize research findings had a substantial influence on observed policy changes, and it was sufficient when combined with a set of enabling conditions that spanned aspects of the environment, institutions, interpersonal relations, and individuals. For the MUDs case, the adoption of MUDs smoking ban policies in jurisdictions outside of Waterloo region (nested TOC) was facilitated by the social and legal environment in Ontario. A trend to condemn smoking in others’ space as both rude and hazardous was codified in laws, and strongly accepted as a norm across initiatives and actors in the province. This environment provided opportunity and motivation for change in Ontario that may not have been present elsewhere. Characteristics of the research also supported adoption. The studies provided findings that were local to Ontario and showed tangible evidence of second-hand smoke drift through multiple units. This in turn brought in incumbent legal concerns about protecting residents and the rights of nonsmokers. The evidence was provided and translated by credible organizations (including Propel), and individual champions were engaged effectively to refine, disseminate, and discuss messages from research with their peers. In combination, these factors were sufficient to support policy change in at least seven of eight regions that adopted smoking bans for MUDs over the 5 years following the original research and policy adoption in Waterloo Region. 2.5.3 Iterative approach applying CA Steps 4, 5, and 6 The CA team met regularly to assess the strengths and limitations of the contribution stories, and identify actions to strengthen them. We used a series of questions to guide our review of each case: How credible is the story? Did the initial assumptions occur to allow the next step in the chain to take place? Does the pattern of observed results validate the results logic and TOC? What are the main weaknesses or gaps in the story? What perspectives may be missing? What about other gaps in data? The iterations helped to uncover a small number of additional data sources, including one new informant for each of two cases, social media statistics, and a list of presentations and consultations that were relevant to two cases. The iterations also helped the CA team experiment with different data displays to examine the strengths and limits of the contribution stories. 3. Summary and reflections The primary purpose of this article was to contribute to dialogue and debate about CA methods. We experimented with CA methods (summarized in Table 4), with a goal of discerning those that were ‘good enough’, that is relevant to the objectives of each CA step, sufficiently rigorous to achieve credible results, and feasible to complete with resources that may be realistic for organizations to invest in CA studies. To our knowledge, assessing CA methods through the lens of getting to ‘good enough’ is a unique contribution in itself, and may be applied to CA methods beyond those that we used. Table 4. Summary of CA methods used in this study CA steps  Application  Team composition and approach  Formed a team with internal (to Propel) and an external member. An iterative, problem-solving approach was used  CA Step 1: Select cases for the study  Defined the study purpose and questions.  Selected cases selected that met three criteria: policy adoption achieved, sufficient information available, and transferability  CA Step 2: Develop an initial TOC for each case  Developed an overall logic model for each case using a common format. The format was modeled after the Bennett hierarchy which itself represents a research-based change theory and used proximal markers of research influence  Chose a nested impact pathway for each case that met three criteria: feasible to complete, previously underexplored links in the research impact pathway, and relevant to Propel strategy and decisions  Synthesized selective social science theories that would inform subsequent CA steps. Developed a theory-based checklist using sources nominated by team members as most highly relevant to the nested impact pathways—what helps or hinders research use in policy and the policy change process  Embellished the nested impact pathways into TOCs by making assumptions between links and contextual influences explicit  CA Step 3: Gather evidence on the TOCs  Identified and gathered relevant quantitative and qualitative evidence from existing documentation and interviews with informants. Evidence gathered met three criteria: relevant, sufficient quality, from diverse sources. Snowball sampling and semi-structured interviews were used for informant interviews and informants nominated other sources of information  CA Steps 4, 5, and 6: Assemble, assess, and strengthen the contribution stories  Assembled the contribution stories using a table of evidence organized by the nested impact pathway. Evidence reported represented main themes and consistency of data across data sources  Assessed strengths and limitations of the contribution stories using three methods: analysis of consistency of data in the TOC; analyses of causal claims using congruence, comparisons, and critical review; and interpreting the contribution stories in relation to scientific theories  Used an iterative approach applying CA Steps 4, 5, and 6. Regular CA team meetings were guided by questions to ensure appropriate application of Steps 4, 5, and 6  CA steps  Application  Team composition and approach  Formed a team with internal (to Propel) and an external member. An iterative, problem-solving approach was used  CA Step 1: Select cases for the study  Defined the study purpose and questions.  Selected cases selected that met three criteria: policy adoption achieved, sufficient information available, and transferability  CA Step 2: Develop an initial TOC for each case  Developed an overall logic model for each case using a common format. The format was modeled after the Bennett hierarchy which itself represents a research-based change theory and used proximal markers of research influence  Chose a nested impact pathway for each case that met three criteria: feasible to complete, previously underexplored links in the research impact pathway, and relevant to Propel strategy and decisions  Synthesized selective social science theories that would inform subsequent CA steps. Developed a theory-based checklist using sources nominated by team members as most highly relevant to the nested impact pathways—what helps or hinders research use in policy and the policy change process  Embellished the nested impact pathways into TOCs by making assumptions between links and contextual influences explicit  CA Step 3: Gather evidence on the TOCs  Identified and gathered relevant quantitative and qualitative evidence from existing documentation and interviews with informants. Evidence gathered met three criteria: relevant, sufficient quality, from diverse sources. Snowball sampling and semi-structured interviews were used for informant interviews and informants nominated other sources of information  CA Steps 4, 5, and 6: Assemble, assess, and strengthen the contribution stories  Assembled the contribution stories using a table of evidence organized by the nested impact pathway. Evidence reported represented main themes and consistency of data across data sources  Assessed strengths and limitations of the contribution stories using three methods: analysis of consistency of data in the TOC; analyses of causal claims using congruence, comparisons, and critical review; and interpreting the contribution stories in relation to scientific theories  Used an iterative approach applying CA Steps 4, 5, and 6. Regular CA team meetings were guided by questions to ensure appropriate application of Steps 4, 5, and 6  Some of the methods we used reinforce the value of what may be considered current good practice for CA. One method is the iterative approach we used when applying all CA steps (Mayne 2008). Our regular team meetings to share work in progress, and discuss what was working well and what could be improved facilitated learning and problem-solving across cases. A second method increasingly suggested by others (Delahais and Toulemonde 2012; Sridharan and Nakaima 2012; Mayne 2015) and successfully used by our team was focusing on nested impact pathways within broader TOCs. Consistent with experiences of others, choosing nested TOCs enabled sufficient depth with data collection, analysis, and interpretation. Third, our data collection strategy reinforced the importance of diversity of types of evidence and perspectives. Our mix of quantitative and qualitative information from a variety of sources, and input from informants with diverse vantage points on the nested TOC (e.g. researchers, knowledge users, policy domain experts) added strength to the contribution stories. Fourth, our efforts to engage with relevant scientific theories when applying CA steps respond to recommendations by others (Willis et al. 2017; Patton 2012). The CA methods we used may also provide some innovations. One innovation is composition of our CA team. Team members brought a mix of necessary and complementary strengths. The Propel in-house team members all had in-depth knowledge of Propel’s mandate, activities, and way of working, which was essential to understand and probe perspectives from informants, both internal and external to Propel. Affiliation with Propel also helped to recruit informants. Another strength of CA team members from Propel is that they were not involved directly in the research and knowledge exchange activities for the three case examples. This helped to maintain appropriate objectivity (and perceptions of it) with the collection and interpretation of information for each case. The external team member also contributed to this objectivity. The complementary mix of relevant content expertise (e.g. tobacco control, policy development), evaluation in general and CA in particular, and project management also contributed to a rigorous and feasible approach. A second innovation was related to selection of cases. The criteria we used and selecting multiple cases to conduct simultaneously both contributed to relevance, rigor, and feasibility. The three criteria we applied for case selection (i.e. policy changes, sufficient information, and transferable lessons) resulted in rich cases to answer our primary question of how Propel activities contributed to changes in tobacco control policies. Although not an explicit criterion, the common focus on tobacco control policy facilitated problem-solving across cases, and also allowed for shared and efficient data collection (e.g. informants who could provide insights on two of the cases in a single interview). Another promising innovation may be the four tasks we used to complete CA Step 2—developing a TOC. Our approach to developing overall logic models (Task 1) enhanced both relevance and rigor in our study. Most helpful were using the Bennett hierarchy for results logic, which allowed us to emphasize stakeholder engagement and participation, and including proximal markers of research influence (see Pasanen and Shaxson 2016) in the impact pathways, which helped to reveal the inherent complexity in how research may influence policy change (Nutley et al. 2007; Boaz et al. 2009). Both of these choices were instrumental in guiding implementation of subsequent tasks in CA Step 2, as well as subsequent CA steps. With respect to Task 2—choosing a nested impact pathway—relevance and feasibility were strengthened by the criteria we used. Feasibility was an explicit criterion, and potential contributions to new knowledge and decision-making ensured scientific and practical relevance. Our Task 3 was perhaps the most innovative in this step. Engaging with scientific theories to inform Task 4 and other CA steps enhanced the rigor in our study. At the same time, we needed to bound this task to ensure it was practical to complete. The subset of theories we chose was very useful in adding theory-based assumptions and contextual influences to the nested impact pathways. Additional value was realized in later CA steps, as described below. Finally, our team explored innovative methods for assembling evidence and assessing the strength of contribution claims. With respect to assembling evidence, our most promising data display for CA Step 4 was the ‘evidence of impact’ table, adapted from others (Montague and Valentim 2010; Morton 2015). The systematic approach to selecting and presenting evidence in this table helped to identify new data to collect (CA Step 5) and to assess the strength of the contribution stories (CA Step 6). The three methods we used to assess strengths and limitations of the contribution stories—analysis of consistency of data; analyses of causal claims using congruence, comparisons, and critical review; and analysis of the contribution stories in relation to scientific theories—all included some innovation and hold promise for application in other CA studies. The three methods were relevant to the goal of CA Step 6, promoted rigor in our assessments, and were feasible to complete. An innovation in the analysis of consistency of data was explicit criteria for different degrees of consistency (e.g. triangulation of evidence from multiple sources and perspectives as most consistent, and data that were contradictory and/or had significant limitations or gaps as least consistent). Perhaps one of the most promising innovations, especially for highly context-sensitive phenomenon such as policy change, may be our mapping of contextual influences along the impact pathway using Pawson’s (2013) categories of contextual factors. This was an initial step toward accounting for influencing factors and identifying possible alternative explanations, both identified as important areas for CA elaborations (Dybdal et al. 2011; Lemire et al. 2012). 3.1 Limitations An overarching limitation was use of retrospective cases. Retrospective cases meant that data sources included historical recollections and written documentation that was not necessarily aligned with the purposes of the study. Understandably, this resulted in incomplete and potentially inaccurate data, and precluded any comparative analysis over time except as it was (inconsistently) historically recorded or recalled. A prospective approach—laying out anticipated theories of change and monitoring progress against them as they unfold—would allow more rigorous and complete explorations and explanations of influence over time (Morton 2015). This would involve ongoing tracking in areas such as reach, engagement, and early reaction of key groups involved in particular impact pathways. Relevant policy domains could also be tracked over a longer period using a prospective approach. This is especially important in population and public health to allow sufficient time for impacts to manifest (Biggs et al. 2014). 3.2 Possible uses of our findings At the outset, we identified three audiences for the CA applications reported in this article. Those with an interest in how research can be used effectively to influence policy processes and outcomes may benefit from insights related to use of theory. One insight is the potential value of using both program (TOC) and scientific theories for planning how research may be used to influence policy change. Our experience also reinforces the importance of integrating theories of engagement, policy change, and spread of innovations for examining the impact of applied health research on the policy process and outcomes. Those with an interest in evaluating research impacts on policy change may benefit from applying and extending the CA methods we used. Those with an interest in applying CA to other policy domains (e.g. beyond health and policies other than government regulations), and different stages of the policy process (e.g. policy implementation, enforcement) may also apply our CA methods, and explore their transferability. 4. Conclusion CA is a promising response to the challenge of understanding and improving the influence of research on policy, at least in the public health domain and for regulatory approaches to tobacco control. By applying CA retrospectively to three cases of tobacco control policy influenced by a Canadian research center (Propel), we offer methods that are sufficiently relevant, rigorous, and feasible—‘good enough’. Getting to ‘good enough’ required careful selection of nested theories of change; strategic use of social science theories, and quantitative and qualitative data from diverse sources; and complementary methods to assemble and analyze evidence for testing the nested theories of change. CA applications will be strengthened further by prospective data collection. Our experience may inform efforts to influence policy with research, to evaluate research impacts on policy using CA, and to apply CA more broadly. Acknowledgements The authors gratefully acknowledge the participation of all informants in the three policy case examples. The authors also acknowledge contributions made by Alyssa Zarnke to developmental work related to the FT case. Funding This work was supported by the Canadian Cancer Society [2011-701019]. Contributions by Cameron D. Willis were supported by The Australian Prevention Partnership Centre through the National Health and Medical Research Council Partnership Centre grant scheme [GNT9100001] with the Australian Government Department of Health, the New South Wales Ministry of Health, Australian Capital Territory Health, Hospitals Contribution Fund (HCF), and the HCF Research Foundation. Cameron D. Willis is supported by a National Health and Medical Research Council Sidney Sax Fellowship [1013165]. References Bennett C., Rockwell K. ( 1995) Targeting Outcomes of Programs (TOP): An Integrated Approach to Planning and Evaluation . Lincoln, NE: University of Nebraska. Biggs J. S. et al.   ( 2014) ‘ A Practical Example of Contribution Analysis to a Public Health Intervention’, Evaluation , 20 / 2: 214– 29. Google Scholar CrossRef Search ADS   Boaz A., Fitzpatrick S., Shaw B. ( 2009) ‘ Assessing the Impact of Research on Policy: A Literature Review’, Science and Public Policy , 36/ 4: 255– 70. Google Scholar CrossRef Search ADS   Buckley A. P. ( 2016) ‘ Using Contribution Analysis to Evaluate Small & Medium Enterprise Support Policy’, Evaluation , 22/ 2: 129– 48. Google Scholar CrossRef Search ADS   Cancer Research UK ( 2015) Worldwide Cancer Incidence Statistics . London: Cancer Research UK. Corbin J., Strauss A. ( 2008) Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 3rd ed. Thousand Oaks, CA: Sage Publications. Google Scholar CrossRef Search ADS   Dauphinee W. D. ( 2015) ‘ The Role of Theory-Based Outcome Frameworks in Program Evaluation: Considering the Case of Contribution Analysis’, Medical Teacher , 37/ 11: 979– 82. Google Scholar CrossRef Search ADS PubMed  Delahais T., Toulemonde J. ( 2012) ‘ Applying Contribution Analysis: Lessons from Five Years of Practice’, Evaluation , 18/ 3: 281– 93. Google Scholar CrossRef Search ADS   Dybdal L., Steffen B. N., Lemire S. ( 2011) ′ Contribution Analysis Applied: Reflections on Scope and Methodology′, The Canadian Journal of Program Evaluation , 25/ 2: 29– 57. Funnell S. C., Rogers P. J. ( 2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models . San Francisco: Jossey-Bass (Wiley). Greenhalgh T. et al.   ( 2016) ‘ Research Impact: A Narrative Review’, BMC Medicine , 14: Guthrie S., et al.   ( 2013) Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Santa Monica, CA: RAND Corporation. Hanney S., Packwood T., Buxton M. ( 2000) ‘ Evaluating the benefits from health research and development centres’, Evaluation , 6/ 2: 137– 60. Google Scholar CrossRef Search ADS   Kingdon J. W. ( 2003) Agendas, Alternatives, and Public Places . New York: Longman. Kok M. O., Schuit A. J. ( 2012) ‘ Contribution Mapping: A Method for Mapping the Contribution of Research to Enhance Its Impact’, Health Research Policy and Systems , 10: 21. Google Scholar CrossRef Search ADS PubMed  Leeuw F. L. ( 2012) ‘ Linking Theory-Based Evaluation and Contribution Analysis: Three Problems and a Few Solutions’, Evaluation , 18/ 3: 348– 63. Google Scholar CrossRef Search ADS   Lemire S. T., Nielsen S. B., Dybdal L. ( 2012) ‘ Making Contribution Analysis Work: A Practical Framework for Handling Influencing Factors and Alternative Explanations’, Evaluation , 18/ 3: 294– 309. Google Scholar CrossRef Search ADS   LSE Public Policy Group ( 2011) Maximizing the Impacts of Your Research: A Handbook for Social Scientists. London: LSE Public Policy Group. Mark M., Henry G. ( 2004) ‘ The Mechanisms and Outcomes of Evaluation Influence', Evaluation ’, 10/ 1: 35– 57. Google Scholar CrossRef Search ADS   Mayne J. ( 2008) ‘ Contribution Analysis: An Approach to Exploring Cause and Effect’, The Institutional Learning and Change (ILAC) Initiative . ILAC Brief 16. Mayne J. ( 2012) ′ Contribution Analysis: Coming of Age?′, Evaluation , 18/ 3: 270– 80. Google Scholar CrossRef Search ADS   Mayne J. ( 2015) ′ Useful Theory of Change Models′, Canadian Journal of Program Evaluation , 30/ 2: 119– 42. Google Scholar CrossRef Search ADS   Mertens D. M., Hesse-Biber S. ( 2013) ‘ Mixed Methods and Credibility of Evidence in Evaluation’, New Directions for Evaluation , 2013/ 138: 5– 13. Google Scholar CrossRef Search ADS   Milat A. J., Bauman A. E., Redman S. ( 2015) ‘ A Narrative Review of Research Impact Assessment Models and Methods’, Health Research Policy and Systems , 13: 7. Google Scholar CrossRef Search ADS PubMed  Montague S. ( 2015) ′The Need to Build Reach Into Results Logic, Theories of Change and Performance Frameworks′, in CES 2015 Annual Conference, Montreal. Montague S., Valentim R. ( 2010) ‘ Evaluation of RT&D: From ‘Prescriptions for Justifying’ to ‘User-Oriented Guidance for Learning’’, Research Evaluation , 19/ 4: 251– 61. Google Scholar CrossRef Search ADS   Morton S. ( 2015) ‘ Research Impact Assessment: A ′Contributions′ Approach’, Research Evaluation , 24/ 4: 405– 19. Google Scholar CrossRef Search ADS   Newcomer K., Brass C. T. ( 2016) ‘ Forging a Strategic and Comprehensive Approach to Evaluation Within Public and Nonprofit Organizations. Integrating Measurement and Analytics Within Evaluation’, American Journal of Evaluation , 37/ 1: 80– 99. Google Scholar CrossRef Search ADS   Nutley S. M., Walter I., Davies H. T. O. ( 2007) How Research can Inform Public Services . Bristol, UK: Policy Press, 376. Google Scholar CrossRef Search ADS   Pasanen T., Shaxson L. ( 2016) How to Design a Monitoring and Evaluation Framework for a Policy Research Project [online text], Overseas Development Institute. Patton M. Q. ( 2012) ‘ A Utilization-Focused Approach to Contribution Analysis’, Evaluation , 18/ 3: 364– 77. Google Scholar CrossRef Search ADS   Pawson R. ( 2013) The Science of Evaluation: A Realist Manifesto . Thousand Oaks, CA: Sage. Google Scholar CrossRef Search ADS   Penfield T. et al.   ( 2014) ‘ Assessment, Evaluations, and Definitions of Research Impact: A Review’, Research Evaluation , 23/ 1: 21– 32. Google Scholar CrossRef Search ADS   Rogers E. M. ( 1995) Diffusion of Innovations , 4th edn. New York: The Free Press. Rogers P. ( 2008) ‘ Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions’, Evaluation , 14/ 1: 29– 48. Google Scholar CrossRef Search ADS   Sridharan S., Nakaima A. ( 2012) ‘ Towards an Evidence Base of Theory-Driven Evaluations: Some Questions for Proponents of Theory-Driven Evaluation’, Evaluation , 18/ 3: 378– 95. Google Scholar CrossRef Search ADS   Stachowiak S. ( 2013) Pathways for Change: 10 Theories to Inform Advocacy and Policy Change Efforts. Seattle: ORS Impact; Washington, DC: The Center for Evaluation Innovation. Willis C. D., Riley B., Stockton L., Viehbeck S., Wutzke S., Frank J. ( 2017) ‘ Evaluating the impact of applied prevention research centres: results from a modified Delphi approach’, Research Evaluation , 26/ 2: 78– 90. Google Scholar CrossRef Search ADS   Wimbush E., Montague S., Mulherin T. ( 2012) ‘ Applications of Contribution Analysis to Outcome Planning and Impact Evaluation’, Evaluation , 18/ 3: 310– 29. Google Scholar CrossRef Search ADS   © The Author 2017. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Research Evaluation Oxford University Press

Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’

Loading next page...
 
/lp/ou_press/using-contribution-analysis-to-evaluate-the-impacts-of-research-on-GefIUXedzb
Publisher
Oxford University Press
Copyright
© The Author 2017. Published by Oxford University Press.
ISSN
0958-2029
eISSN
1471-5449
D.O.I.
10.1093/reseval/rvx037
Publisher site
See Article on Publisher Site

Abstract

Abstract Assessing societal impacts of research is more difficult than assessing advances in knowledge. Methods to evaluate research impact on policy processes and outcomes are especially underdeveloped, and are needed to optimize the influence of research on policy for addressing complex issues such as chronic diseases. Contribution analysis (CA), a theory-based approach to evaluation, holds promise under these conditions of complexity. Yet applications of CA for this purpose are limited, and methods are needed to strengthen contribution claims and ensure CA is practical to implement. This article reports the experience of a public health research center in Canada that applied CA to evaluate the impacts of its research on policy changes. The main goal was to experiment with methods that were relevant to CA objectives, sufficiently rigorous for making credible claims, and feasible. Methods were ‘good enough’ if they achieved all three attributes. Three cases on government policy in tobacco control were examined: creation of smoke-free multiunit dwellings, creation of smoke-free outdoor spaces, and regulation of flavored tobacco products. Getting to ‘good enough’ required careful selection of nested theories of change; strategic use of social science theories, as well as quantitative and qualitative data from diverse sources; and complementary methods to assemble and analyze evidence for testing the nested theories of change. Some methods reinforced existing good practice standards for CA, and others were adaptations or extensions of them. Our experience may inform efforts to influence policy with research, evaluate research impacts on policy using CA, and apply CA more broadly. 1. Introduction Interest in research evaluation is increasing for many reasons, to inform action, learning, accountability, and funding allocation decisions (Guthrie 2013; Penfield et al. 2014). To achieve these different purposes, there are diverse efforts to examine and document research impact (Penfield et al. 2014). Across these efforts, there is most consistency on measuring academic impacts, intellectual contributions made by research to a particular field of study (Greenhalgh et al. 2016). Measuring ‘societal’ impacts is less consistent and more challenging (Penfield et al. 2014; Greenhalgh et al. 2016). The challenge is identifying causal pathways that clearly link research to health, social, or economic changes (Hanney et al. 2000). Identifying and understanding these pathways are especially difficult for applied research efforts related to public policymaking, for which the links between research and impact are influenced by a wide array of factors and players (Milat et al. 2015; Greenhalgh et al. 2016). The importance of overcoming this challenge was reinforced by a modified Delphi study completed by members of the author team (Willis et al. 2017). We explored indicators that may be important for evaluating practical impacts of public health research centers, and identified promising directions for development of new methods for assessing these indicators. Consistent with recommendations by others (Boaz et al. 2009; Milat et al. 2015), panelists generally agreed that a critical focus for public health research centers is to influence policy development, implementation, and outcomes, and that current methods for capturing the influence of research on policy are underdeveloped. Theory-based evaluations are proposed as particularly promising, with considerable interest in contribution analysis (CA) (Dauphinee 2015; Buckley 2016). 1.1 Using CA for assessing research impact on policy Most scientific inquiry focuses on questions of attribution. Yet attribution analysis tends to focus on direct, verifiable causality that is not aligned with a generally accepted view of the policy process involving complex interconnections among activities and observed outcomes (Patton, 2012). CA, in contrast, can credibly assess cause and effect relationships in circumstances when impacts result from a complex interplay of actions (including research) by multiple players, and a variety of contextual factors (Mayne 2008). Application of CA involves six steps, starting with identifying the attribution problem to be addressed (Step 1), developing a theory of change (TOC) (Step 2), gathering evidence on the TOC (Step 3), assembling evidence and assessing the contribution story (Step 4) through an iterative process of seeking additional evidence (Step 5), and revising and strengthening the contribution story (Step 6) (Mayne 2008, 2012, 2015). CA is generally viewed as conceptually appealing, and yet methods for implementing the six steps to optimize the credibility and utility of CA are not well developed (Dybdal et al. 2011; Lemire et al. 2012; Wimbush et al. 2012; Dauphinee 2015). As a result, several authors point to methodological elaborations that may strengthen applications of CA and theory-based evaluations more generally (Delahais and Toulemonde 2012; Kok and Schuit 2012; Leeuw 2012; Lemire et al. 2012; Sridharan and Nakaima 2012; Willis et al. 2017; Morton 2015). Proposed elaborations address several questions the evaluation field is grappling with, such as: How do we go about strengthening theories of change that incorporate concepts of reach? (Montague 2015) How can we build in varied assumptions about pathways of impact, especially how these are influenced over time? (Mayne 2015) How might we bound analyses (e.g. focus on nested contribution stories) so they are both rigorous and feasible? (Mayne 2015) How can we strengthen the use of scientific theories in CA and other theory-based evaluations? (Willis et al. 2017) How can we account for competing explanations and influencing factors? (Dybdal et al. 2011) 1.2 Study purpose and significance As part of an organizational-level evaluation, our team applied CA to evaluate the impacts of research on a specific domain of public health policy—government regulations for tobacco control in Canada, spanning municipal, provincial, and federal levels. We used three cases (described below) from the Propel Centre for Population Health Impact (Propel), a national research program of the Canadian Cancer Society, and hosted by the Faculty of Applied Health Sciences at the University of Waterloo, Ontario, Canada. Propel exists to prevent cancer and other chronic diseases and their shared behavioral and environmental causes, and achieves its mandate by catalyzing and conducting relevant studies using the most appropriate and rigorous methods, and moving evidence into action. Research and knowledge exchange activities completed by Propel are intended to contribute to improvements in policies and programs that impact health at a population level. Tobacco control policies are an important case in point, and have been a focus within Propel for 25 years. Propel activities are part of an increasingly diverse and complex array of actions by multiple sectors and players, interacting with a similarly diverse set of contextual factors, all influencing policy development, implementation, and outcomes. These conditions are consistent with circumstances under which CA is considered particularly useful. The primary purpose of this article is to advance CA methods. Our goal was to use methods that would be relevant to CA objectives, sufficiently rigorous to make defensible contribution claims, and also practical to implement as part of an organizational approach to evaluation (Newcomer and Brass 2016). Achieving these attributes is how we defined a ‘good enough’ CA. This article may be of interest to those concerned with how research can be used effectively to influence policy processes and outcomes. It may also be of interest to those wanting to evaluate research impacts on policy change, including funders, evaluators, and knowledge users. A third audience is those who are interested in applying CA, especially for evaluating research impacts on policy change, and especially in the domain of population and public health. The experience reported in these specific domains, however, may inform other uses of CA. 2. Application of CA We applied CA to evaluate the influence of Propel initiatives on the development and uptake of three areas of tobacco control policy in Canada: creation of smoke-free multiunit dwellings (MUDs), creation of smoke-free outdoor spaces (SFOS), and regulation of flavored tobacco products (FT). Following a description of the evaluation team, we describe our application of CA to the three cases, and use one case (MUDs) as an illustrative example. 2.1 CA team and approach We used a team approach, with members from inside and outside Propel. Internal team members included faculty (B. R., J. Y., C. W.) and research staff (A. K., L. S.), all of whom have in-depth knowledge of Propel’s mandate, activities, and way of working, yet who were not directly involved in the research and knowledge exchange activities for the three tobacco control policy case examples. Propel internal team members brought content expertise in tobacco control and skills in evaluation and project management. They also had access to people who participated in each of the three cases, mainly through other Propel staff and their collaborators. The external team member (S. M.) is a consultant with extensive expertise in evaluation, including CA. Team roles included an overall scientific lead for the study (B. R.), a lead for each case example (A. K. for two cases and S. M. for one), CA methodological guidance and experience (S. M., J. Y.), and linkages to other center evaluation activities (B. R., L. S., C. W.). We conducted these studies over a 1-year period. Estimated time commitments from members of the CA team were: 0.5 full-time equivalents (FTEs) for in-house Propel staff, 0.2 FTE for in-house faculty, and 0.15 FTE for the external consultant. Our team complied with the recommended iterative approach when applying the CA steps (Mayne 2008). For all steps, we held regular team meetings to discuss what was working well and any existing issues or obstacles. Problem-solving occurred within and across the three cases. 2.2 Select cases for the study (CA Step 1) CA’s first step is to set out the attribution problem to be addressed. In our application, this happened in two stages. The first was selection of cases that would allow us to address the overall study purpose of examining how Propel initiatives influenced the adoption of policy, and the second was identifying the nested theories of change that we would focus on within each case. The latter was done after partial implementation of CA Step 2, and is described in that section. Our goal was to select a small number of cases (2 or 3), and each needed to meet three criteria. First, evidence of policy adoption was needed, in at least one jurisdiction and preferably more. Second, sufficient information related to Propel contributions and other influencing factors needed to exist or could be collected. This was done by reviewing organizational records (a central tracking system) for relevant documents, and consulting with Propel personnel directly involved in each case to explore access to collaborators outside of Propel. Third, circumstances and events related to the case could be expected to apply to other policy domains and other jurisdictions. These assessments were done by team members, all with relevant expertise and experience. By applying these criteria, three cases were identified. MUDs are a set of tobacco control policies adopted by regional government agencies, individual landlords, and related policymakers to regulate smoking in dwellings with multiple living units (e.g. apartments). SFOS is a set of tobacco control policies at municipal and provincial levels designed to protect people in a variety of outdoor spaces (e.g. parks, playgrounds, patios) from exposure to second-hand smoke, and also to minimize the social modeling of smoking, particularly among children and youth. FTproduct regulations are under provincial and federal jurisdictions, and emerged as an important policy domain in 2010, especially to protect youth from initiation and use of new tobacco products, including menthol. 2.3 Develop an initial TOC for each case (CA Step 2) A TOC represents hypothesized cause and effect relationships and related assumptions within an overall pathway of change. Theories of change can be shown at various levels of detail, with general agreement among CA scholars to begin with a high-level TOC, before progressively adding detail as the model develops. We applied the principle of taking a progressive and iterative approach to developing the TOCs that would become the primary focus of our CA. We undertook four main tasks that are described below. First, we developed an overall logic model for each case. We used a common format across cases (consistent with Mayne, 2015) based on the Bennett hierarchy with seven interdependent components (Bennett and Rockwell 1995). Guided by Pasanen and Shaxson (2016), and as recommended for complex processes (Nutley et al. 2007; Boaz et al. 2009), we also included proximal markers of research influence on policy change (e.g. research products, knowledge exchange activities). The initial logic models were completed by the CA team in consultation with Propel personnel directly involved in each case (minimally, the lead faculty and staff members). Propel personnel were asked to identify documents that provided an overview of Propel activities, participating individuals and organizations, and policy-related outcomes; participated in at least one in-depth conversation with the CA team; and provided feedback on the logic model. Figure 1 shows the product of this first task for the MUDs case. Figure 1. View largeDownload slide Overall logic model for the creation of smoke-free MUDs. Figure 1. View largeDownload slide Overall logic model for the creation of smoke-free MUDs. As shown in Fig. 1, creating smoke-free MUDs had three main areas of development: early pilot work in the Waterloo region to support policy development (Section A); Propel’s research to increase understanding of policy implementation (Section B); and knowledge exchange and policy change processes (Section C) within Waterloo and dissemination to other communities. These three main developments were underpinned by shared inputs, including financial and personnel investments from Propel, a collaborative approach to research that actively engaged knowledge users, strong existing relationships between Propel researchers and local public health colleagues, and synthesis of research information in meaningful ways for knowledge users. Moving from left to right, the logic model includes a temporal dimension that shows the progression of activities and outcomes over time. Logic models for the other two cases were similar, with each including feedback loops showing how early products, results, and learnings informed subsequent work. A difference in the logic models across cases was the level of complexity. The FT case was less complex than the other cases, mostly due to a more limited set of stakeholders involved. FT policy changes were at federal and provincial levels, and provincial advocates (as direct links to policymakers) were the main stakeholders involved in the uptake of Propel products. In the other two cases, the policy process involved more diverse groups of stakeholders, and impact spread across more settings and contexts. The SFOS case, for example, involved policy change in several municipalities initially, followed by changes in provincial policy. As recommended by others (Delahis and Toulemonde 2012; Sridharan and Nakaima 2012; Mayne 2015), our second task was to choose a nested impact pathway within each comprehensive logic model. The nested pathway would become the focus of subsequent steps in the CA. Nested pathways were selected that would be feasible to complete, and that represented previously underexplored links in the research impact pathway with high relevance to Propel strategy. Nested pathways, therefore, would contribute to new knowledge, and provide useful information to guide future efforts by Propel and others to influence policy with research. Across all three cases, the pathway selected for further analysis focused on Propel’s role in knowledge translation and dissemination, and the influence of these activities on the engagement of key audiences, changes in capacity, and the adoption of relevant smoke-free policies. For the MUDs case, the nested impact pathway is shown Fig. 1. The third task within CA Step 2 was to synthesize aspects of relevant social science theories that could inform subsequent CA steps. This task was intended to address a common criticism of limited explicit use of scientific theories in CA and evaluation generally (Patton 2012; Willis et al. 2017). Our goal was to engage with scientific theories in ways that added value and credibility to our studies, while also ensuring feasibility for our own and other CA studies. Aiming to strike this balance, and acknowledging a very large and diverse set of potentially relevant theories (cf. Stachowiak 2013), each CA team member nominated a small subset of theories they considered to be most relevant to the nested impact pathways. We specifically identified theories that focus on what helps and hinders research use in policy, and the policy change process. Using nominated sources, we developed a theory-based checklist with three sections. The first section refers to general parameters for the influence of research, including various environments, timeframes, intentions, and levels within which research influence may occur. The second section refers to mechanisms of research influence (e.g. priming, persuasion, elaboration, etc.) and is primarily informed by Mark and Henry (2004), Pasanen and Shaxson (2016), and Pawson (2013). The third section of the checklist refers to contextual factors, all of which are known to influence research utilization, and include characteristics and capacities of people, their relationships and the wider environment. The contextual factors are mostly informed by Pawson’s (2013) four I’s of contextual influence (i.e. individuals, interpersonal relations, institutional settings, and infrastructure), Kingdon’s (2003) advocacy and policy change contexts, and Rogers’ (1995) characteristics of innovations that support diffusion. The fourth and final task in CA Step 2 was to embellish the nested impact pathways into theories of change, by naming a subset of assumptions and contextual factors that may be most influential in explaining the links in the impact pathway for each case. These theories of change were the focus of subsequent CA steps. The TOC for the MUDs nested impact pathway is shown in Fig. 2. Figure 2. View largeDownload slide TOC for the MUDs nested impact pathway (adapted from Mayne, 2015). Figure 2. View largeDownload slide TOC for the MUDs nested impact pathway (adapted from Mayne, 2015). 2.4 Gather evidence on the TOCs (CA Step 3) The primary purpose of CA Step 3 is to test the validity of the TOC (observed results, assumptions, influencing factors). Our goal was to gather the most relevant evidence, all of sufficient quality and from diverse sources to test each TOC. Relevance and sufficient quality were minimum criteria for all sources of evidence. Seeking diverse sources of evidence respected the inherent complexity of our cases (e.g. many actors and agencies involved in the policy development process over time) (Rogers 2008), and also responded to a criticism that most studies rely on interviews with principal investigators and/or peer review to assess research impacts (Biggs et al 2014; Penfield et al 2014; Milat et al 2015). One way diversity was achieved was using mixed methods (Mertens and Hesse-Biber 2013), allowing us to explore the extent to which quantitative (numerical) and qualitative (text) information was consistent and/or complementary for different parts of the TOC. Diversity was also achieved by interviewing people who had different roles in the policy process (e.g. researchers, knowledge users, knowledgeable commentators in relevant policy domains), and thereby different vantage points on the context, process, and outcomes for each TOC. Perspectives external to Propel were given particular attention, as sources of information that broadened and increased the robustness of the evidence to either support contribution assertions or provide alternative explanations. A practical aim was to gather evidence as efficiently as possible. To this end, we sought to identify existing information (collected for a purpose other than the CA) that met our criteria, and also collected new information to fill gaps and expand on existing evidence. Sources of evidence were identified progressively. For each case, we started data collection by interviewing up to three personnel from Propel who were most directly involved in each of the three cases. In addition to collecting primary data from these personnel (interview strategy described in more detail below), we asked these informants to identify other sources of information, including additional informants (i.e. snowball sampling; cf. Corbin and Strauss 2008), documents, Web-based statistics, media, etc. When seeking nominations for additional sources of information, we emphasized the importance of diverse types of evidence and diverse perspectives, as described above. To illustrate the mix of evidence, Table 1 shows the sources of evidence for the MUDs case. Table 1. Sources of evidence for the MUDs case Data sources  Description  Reports and videos  Twenty-six documented examples of smoke-free MUDs activities and initiatives, e.g. Waterloo Region Smoke-Free Housing Policy Evaluation report  Interviews  Consultations with 12 informants, e.g. advocates, researchers, and public health staff from multiple jurisdictions  Tracking forms  Detailed documentation of over 60 consultations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions, and the status of subsequent smoke-free policy adoption over time (from 2009 to 2015)  Web statistics  Downloads and shares (variable for this time period)  Citation analysis  Some citation evidence from public health reports  Data sources  Description  Reports and videos  Twenty-six documented examples of smoke-free MUDs activities and initiatives, e.g. Waterloo Region Smoke-Free Housing Policy Evaluation report  Interviews  Consultations with 12 informants, e.g. advocates, researchers, and public health staff from multiple jurisdictions  Tracking forms  Detailed documentation of over 60 consultations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions, and the status of subsequent smoke-free policy adoption over time (from 2009 to 2015)  Web statistics  Downloads and shares (variable for this time period)  Citation analysis  Some citation evidence from public health reports  Each of the three cases differed in the type and amount of existing information available. Evidence that was common across cases included websites, attendance lists, citations, media coverage, and social media analytics. Some unique sources were also used, such as records kept by Region of Waterloo of presentations and consultations undertaken with municipalities and other stakeholders interested in smoke-free MUDs, and a publicly available inventory of SFOS policies across Canada. The amount of data from different sources also varied across cases, with the most comprehensive documentation for media coverage on FT. Common limitations to existing evidence were incomplete information on the reach and uptake of Propel documents, outcomes from partner meetings, audiences for presentations, and social media statistics. Some of these data were unavailable, inconsistently recorded, or untraceable (e.g. Propel documents or videos were often downloaded from links on partner’s sites and those statistics were not reflected in the Propel data). Across all cases, informants internal and external to Propel were interviewed, with variable numbers and diversity of informants. In the FT case, informants were fewer and came from less diverse settings compared to those from the MUDs and SFOS cases. Most FT case informants were researchers and tobacco control advocates at provincial and national levels, many of whom are part of an existing pan-Canadian network for implementing a national survey and using data from this survey to inform advocacy. In contrast, a larger number and more diverse set of informants were interviewed for the MUDs and SFOS cases. Main informants included researchers from within and beyond Propel, local public health practitioners from Waterloo and other communities, and nongovernment organizations and tobacco control advocates at provincial and national levels. Similar to the FT case, some of the informants were members of an existing network. Several informants were consistent across the MUDs and SFOS cases, so when possible and appropriate, questions on both cases were completed in a single interview. Interviews for each case were terminated when all nominated informants had been contacted. We used semi-structured interviews (Corbin and Strauss 2008) that were informed by the TOCs and the theory-based checklist. Interview questions and probes were customized for each case and allowed informants to tell their story of the events leading up to policy change, including factors that were particularly relevant in their context. In the MUDs case, for example, questions contextualized the influence of Propel’s work in relation to current activities in Canada focused on smoke-free policies in MUDs, the contributions of other groups, and what policy change may have occurred in the absence of Propel’s work. Processes of influence were also explored, such as the degree to which Propel’s work may have achieved influence through priming (e.g. increasing the salience of a concept, such as everyone has the right to breathe clean air), or through capacity building by improving understanding of issues related to MUDs. Informants were generally a rich source of information, and they provided consistent feedback on Propel’s contributions to policy change. As expected, recall was better when research and policy change occurred more recently (such as FT policy change), and from informants directly involved in the events. 2.5 Assemble, assess, and strengthen the contribution stories (CA Steps 4, 5, and 6) CA Steps 4 through 6 were performed iteratively, beginning with assembling the evidence and assessing its strengths and limitations (CA Step 4), and then completing cycles of gathering additional evidence (CA Step 5) and revising and strengthening the contribution story (CA Step 6). We combine the three steps in this section, since additional data gathering (Step 5) and techniques for assessing strength of the contribution stories (Steps 4 and 6) were repeated in the iterations. We describe how we assembled the contribution stories, assessed their strengths and limitations, and applied Steps 4 through 6 iteratively. 2.5.1 Assembling the contribution stories We adapted approaches used by others (Montague and Valentim 2010; Morton 2015) for assembling evidence. Our goals were to optimize accuracy and sufficiency of data assembly as a basis for subsequent assessments, while also making sure displays were practical to complete. A first display was intended to facilitate a rigorous review of the strengths and limits of the emerging contribution stories. Building on Morton’s ‘evidence of impact’ table (Morton 2015: 411), evidence was assembled for the TOC components: activities in the nested impact pathway, assumptions linking these, and contextual influences. Limitations to the evidence were also identified (see Table 2). Table 2. Illustrative evidence for the smoke-free MUDs TOC Impact pathway  Illustrative evidence   Limitations of the evidence  Activities  Context    Advocacy intermediaries and advocates are engaged  Reports and videos—Documentation of smoke-free MUDs policy change in Waterloo was distributed through channels, e.g. provincial knowledge exchange forum Interviews—Informant assertions of the uptake of Propel messaging. One advocate said, ‘Public health pays a lot of attention to what comes out of Propel … products are consumable by the government and public.’ Tracking form—Documentation of 60+ presentations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions  Interviews—Informant assertions of several factors that influenced engagement, e.g. Propel’s consultative and participative approach, the accessibility and credibility of the products, and the tangibility of the evidence  Detailed recall was not consistent across all informants, as some people could not remember who was involved in what    Decision/policymakers in other communities receive information on second-hand smoke policies in Waterloo MUDs and engage in briefings  Interviews—Informant assertions, e.g. ‘Everybody in MUDs is using our process and our documentation as a template.’ (public health staff)  Interviews—Informant assertions that coordinated messaging was influential in getting information to decision-makers  Assumptions about the conditions needed to engage decision/policymakers are based on the recall of a selective set of participants    Reach assumptions: Information and products are accessible  Reports and videos and Web statistics—A range of stakeholders were exposed to the information through multiple channels and mediums Interviews—Informants discussed the influence of Propel’s knowledge translation stating, ‘Accessible information products are important, It is not enough for science to be published in a journal and wait for the attention to follow.’ (advocate)  Web statistics do not show a comprehensive picture of reach as reports and videos uploaded from partner sites were not included in Propel’s data    Constructive reactions  Interviews—Numerous informants noted the positive reaction of groups to Propel scientists, staff, and presentations  Interviews—Informants referred to an environment that promoted constructive reactions, e.g. ‘People are more likely to be persuaded if they have a face or a name.’ (advocate)  Retroactively, a survey was not done, as a robust source of evidence    Capacity change assumptions: Rationale and need for MUDs smoking ban is understood Smoking ban is perceived as achievable  Reports and videos—Propel’s research showed the impact of smoke-free MUDs policy on tenant perceptions, attitudes, and self-assessed smoking behavior Interviews—Informants asserted that this research was ‘useful to them to be able to make an informed decision on where they wanted to go.’ (researcher) Informants also discussed the influence of Waterloo’s work on other jurisdictions, e.g. ‘The fact that the change has been demonstrated by someone first was viewed as important in building confidence about the do-ability of MUDs.’ (former community of practice staff)  There was insufficient research evidence to determine the economic impacts of the MUDs policy, e.g. on regional workload and costs    Knowledge gain and commitment to reducing second-hand smoke in MUDs through smoking bans  Interviews—Informants referred to the Waterloo experience as a motivator for change saying, ‘If it had not happened in Waterloo, it would not have happened in [name of community].’ (researcher)  Interviews—Informant assertions that successful policy change by the Region of Waterloo built commitment  Without a comparison group, it is not possible to be unequivocal about the influence of Propel    Behavior change assumptions: Waterloo policies are appropriate or can be adapted in other communities Second-hand smoke is a priority for MUDs in other communities Smoking ban is supported by the majority of MUDs residents  Interviews—Informants discussed the positive influence of the policy change in Waterloo Region in paving the way for other jurisdictions, e.g. ‘Waterloo being one of the first in Ontario was obviously a big role model for [our community] to emulate.’ (public health staff)  There is incomplete evidence on the priorities of housing authorities and the level of support for similar policies in other jurisdictions across the province at that time    Smoke-free MUDs policy adoption  Interviews – Informant assertions that Propel work was influential in policy change in other jurisdictions. ‘Propel made it more reasonable, possible for our community.’ (public health unit staff) Tracking form—Direct link between timing of consultations and subsequent policy change in seven Ontario communities  Interviews—Informant assertions that the linked interests, through a regional government committee structure, may have positively influenced smoke-free policy adoption in MUDs  Reflections on other necessary conditions supporting policy adoption in other communities are not available  Impact pathway  Illustrative evidence   Limitations of the evidence  Activities  Context    Advocacy intermediaries and advocates are engaged  Reports and videos—Documentation of smoke-free MUDs policy change in Waterloo was distributed through channels, e.g. provincial knowledge exchange forum Interviews—Informant assertions of the uptake of Propel messaging. One advocate said, ‘Public health pays a lot of attention to what comes out of Propel … products are consumable by the government and public.’ Tracking form—Documentation of 60+ presentations on smoke-free MUDs by Region of Waterloo public health staff to other groups and jurisdictions  Interviews—Informant assertions of several factors that influenced engagement, e.g. Propel’s consultative and participative approach, the accessibility and credibility of the products, and the tangibility of the evidence  Detailed recall was not consistent across all informants, as some people could not remember who was involved in what    Decision/policymakers in other communities receive information on second-hand smoke policies in Waterloo MUDs and engage in briefings  Interviews—Informant assertions, e.g. ‘Everybody in MUDs is using our process and our documentation as a template.’ (public health staff)  Interviews—Informant assertions that coordinated messaging was influential in getting information to decision-makers  Assumptions about the conditions needed to engage decision/policymakers are based on the recall of a selective set of participants    Reach assumptions: Information and products are accessible  Reports and videos and Web statistics—A range of stakeholders were exposed to the information through multiple channels and mediums Interviews—Informants discussed the influence of Propel’s knowledge translation stating, ‘Accessible information products are important, It is not enough for science to be published in a journal and wait for the attention to follow.’ (advocate)  Web statistics do not show a comprehensive picture of reach as reports and videos uploaded from partner sites were not included in Propel’s data    Constructive reactions  Interviews—Numerous informants noted the positive reaction of groups to Propel scientists, staff, and presentations  Interviews—Informants referred to an environment that promoted constructive reactions, e.g. ‘People are more likely to be persuaded if they have a face or a name.’ (advocate)  Retroactively, a survey was not done, as a robust source of evidence    Capacity change assumptions: Rationale and need for MUDs smoking ban is understood Smoking ban is perceived as achievable  Reports and videos—Propel’s research showed the impact of smoke-free MUDs policy on tenant perceptions, attitudes, and self-assessed smoking behavior Interviews—Informants asserted that this research was ‘useful to them to be able to make an informed decision on where they wanted to go.’ (researcher) Informants also discussed the influence of Waterloo’s work on other jurisdictions, e.g. ‘The fact that the change has been demonstrated by someone first was viewed as important in building confidence about the do-ability of MUDs.’ (former community of practice staff)  There was insufficient research evidence to determine the economic impacts of the MUDs policy, e.g. on regional workload and costs    Knowledge gain and commitment to reducing second-hand smoke in MUDs through smoking bans  Interviews—Informants referred to the Waterloo experience as a motivator for change saying, ‘If it had not happened in Waterloo, it would not have happened in [name of community].’ (researcher)  Interviews—Informant assertions that successful policy change by the Region of Waterloo built commitment  Without a comparison group, it is not possible to be unequivocal about the influence of Propel    Behavior change assumptions: Waterloo policies are appropriate or can be adapted in other communities Second-hand smoke is a priority for MUDs in other communities Smoking ban is supported by the majority of MUDs residents  Interviews—Informants discussed the positive influence of the policy change in Waterloo Region in paving the way for other jurisdictions, e.g. ‘Waterloo being one of the first in Ontario was obviously a big role model for [our community] to emulate.’ (public health staff)  There is incomplete evidence on the priorities of housing authorities and the level of support for similar policies in other jurisdictions across the province at that time    Smoke-free MUDs policy adoption  Interviews – Informant assertions that Propel work was influential in policy change in other jurisdictions. ‘Propel made it more reasonable, possible for our community.’ (public health unit staff) Tracking form—Direct link between timing of consultations and subsequent policy change in seven Ontario communities  Interviews—Informant assertions that the linked interests, through a regional government committee structure, may have positively influenced smoke-free policy adoption in MUDs  Reflections on other necessary conditions supporting policy adoption in other communities are not available  2.5.2 Assessing strengths and limitations of the contribution stories The credibility of contribution claims is paramount in CA, and methods to enhance credibility are most underdeveloped. We examined strengths and limitations of the contribution stories using different approaches, with three of the most promising described below. One assessment examined the consistency of data in the evidence of impact tables (Table 2). Consistency was considered highest if findings were similar across multiple sources; medium if findings from different sources were complementary and supporting similar directions of influence in the pathway; and lowest if findings from different sources were contradictory and/or significant gaps or limitations were apparent. Although the specific data and themes varied across cases, each case had evidence from multiple sources and examples of data with different degrees of consistency. This common pattern is shown in Table 2, with the illustrative evidence for the first activity in the MUDs impact pathway. This activity (engagement of advocacy intermediaries and advocates) was supported with evidence from reports and videos, interviews, and the tracking form. Interview data highlighted the importance of characteristics of the evidence (tangibility), Propel’s approach (consultative and participatory), and the products themselves (accessible and credible) that may have influenced the success of the engagement. Since at least 7 years had passed since the early engagement for the MUDs research, detailed and nuanced information (e.g. about the roles and contributions from Propel, and influential contextual factors) for this stage of engagement was limited. A second assessment of strengths and limits was adapted from Funnell and Rogers (2011), and is shown in Table 3. We examined congruence with the program theory, comparisons to what might have happened in the absence of the intervention, and critical review of exceptions to anticipated patterns and other possible explanations. As shown with illustrative evidence in Table 3, evidence for making contribution claims is strong in the MUDs case. The congruence between the TOC and available evidence is high. In the counterfactual comparison, the evidence points to an increased trajectory of smoke-free MUDs across Ontario following Propel’s work to evaluate and disseminate the findings of smoke-free MUDs in Waterloo. In addition, evidence from interviews suggests that informants estimate that the spread of smoke-free MUDs would have been significantly slower without the work of Propel. With respect to critical review, the possible influence of US-based research on second-hand smoke in buildings was probed with knowledge user informants. They noted that local evidence was considered more relevant and had a stronger influence on their actions than research from outside of Canada. The timing of information release to policy change corroborates that assertion. Table 3. Assessing strength of Propel contributions to smoke-free MUDs using congruence, counterfactual comparison, and critical review (adapted from Funnell and Rogers, 2011) Methods and techniques  Illustrative evidence  Congruence   Comparing achievement of sequence of results outcomes: Where the intended outcomes have been achieved, the intermediate outcomes have also been achieved. Where the intended outcomes have not been achieved, the intermediate outcomes have also not been achieved  Tracking form: Region of Waterloo consultations and subsequent policy change in seven Ontario communities follow the expected pattern as detailed in Table 2. The necessary events and conditions needed to influence the policy change were present and aligned with policy change and innovation and diffusion theories   Disaggregating results for complicated interventions: Checking the results matches the TOC when a more complex causal package is important  Details are included in Figure 2 and Table 2   Modus operandi: Some interventions have distinct patterns of effect that can be used as evidence for causal influence  Interviews: Informants asserted that engagement was a key element influencing policy change, and noted Propel’s participation in high engagement activities, e.g. engagement of decision/policymakers was facilitated by the sharing of accessible documentation and process, and coordination across messaging groups Tracking form: Documentation of greater than 60 consultations and presentations by Region of Waterloo staff   Comparing timing of outcomes with program theory: Program theory may predict how long before final outcomes are evident and also whether these are likely to be maintained, increase, or decay over time  Tracking form: Region of Waterloo consultations occurred prior to policy change in seven Ontario communities Interviews: Informants from two communities asserted that engagement with the Waterloo experience directly influenced their adoption of a smoke-free MUDs policy   Comparing dose–response patterns with program theory: Program theory might predict whether increased exposure to an intervention is expected to have a positive, negative, or curvilinear relationship to the intended outcomes  Interviews: Informant assertions that one Ontario community consulted extensively with Waterloo. Notably, that community has the most widespread coverage of housing units included in their smoke-free MUDs policy change   Comparing expert predictions with actual results: For evaluations conducted over a period of time, it is possible to make predictions based on program theory or an emerging theory of wider contributors to outcomes, and then to follow up these predictions over time  N/A   Asking participants: While participants might sometimes have their own reasons for attributing or not attributing changes to an intervention, detailed accounts of their change trajectory can be credible  Interviews: All direct participants consulted in this case study asserted that the Waterloo initiative had influence. There were no participants who did not assert influence   Asking other informants  Interviews: Informants from related associations and exposed communities all asserted Waterloo/Propel influence on their pathway to policy change   Making comparisons across cases: The method of qualitative comparative analysis compares the configurations of different cases to identify the components that produce specific outcomes. Program theory can help to identify the variables that should be included in this analysis  Comparison across cases shows that Waterloo/Propel might be important but not sufficient. Further investigation may reveal that Propel/Waterloo may be necessary but not sufficient for MUDs in Ontario in the present time frame  Counterfactual comparison     Comparing the trajectory before and after the intervention: Time series data can provide a credible estimate of the counterfactual in fairly stable situations  Tracking form: There were no Ontario MUDs prior to the start of the Propel work; however, the Region of Waterloo tracking shows after the Propel work seven communities in Ontario adopted similar smoke-free policies for public housing. Each of these seven communities consulted Waterloo processes/experiences prior to adopting their policies   Thought experiments to develop plausible alternative scenarios: Evidence about policies and procedures and other opportunities in some cases can be used to develop a realistic scenario of the chain of events in the absence of an intervention  Interviews: Informants were asked to speculate what would have happened without the Waterloo/Propel work. No informant suggested that the development of MUDs would have gone the same way without it. Those who estimated suggested that ‘we might not be there yet’ and that the impact was likely extensive, i.e. several years  Critical review     Identifying alternative explanations and seeing if they can be ruled out: Alternative explanations might come from insiders (participants, informants), previous research, or speculation  Interviews: Informants noted that American research revealed an awareness of the issue of second-hand smoke risk in buildings, but that people typically prefer local examples   Identifying and explaining exceptions: Exceptional cases might be successes that were expected to be failures or vice versa. Ideally, an evaluation can explain these, or at least document that they exist, and not lose this information by focusing only on the overall pattern  Tracking form: Among the seven Ontario communities to adopt smoke-free MUDs policies after consultations with Waterloo, one community may be the exception to the rule. This community appears to have changed prior to consultation (albeit they still passed their policy 2 years after Waterloo’s passage)  Methods and techniques  Illustrative evidence  Congruence   Comparing achievement of sequence of results outcomes: Where the intended outcomes have been achieved, the intermediate outcomes have also been achieved. Where the intended outcomes have not been achieved, the intermediate outcomes have also not been achieved  Tracking form: Region of Waterloo consultations and subsequent policy change in seven Ontario communities follow the expected pattern as detailed in Table 2. The necessary events and conditions needed to influence the policy change were present and aligned with policy change and innovation and diffusion theories   Disaggregating results for complicated interventions: Checking the results matches the TOC when a more complex causal package is important  Details are included in Figure 2 and Table 2   Modus operandi: Some interventions have distinct patterns of effect that can be used as evidence for causal influence  Interviews: Informants asserted that engagement was a key element influencing policy change, and noted Propel’s participation in high engagement activities, e.g. engagement of decision/policymakers was facilitated by the sharing of accessible documentation and process, and coordination across messaging groups Tracking form: Documentation of greater than 60 consultations and presentations by Region of Waterloo staff   Comparing timing of outcomes with program theory: Program theory may predict how long before final outcomes are evident and also whether these are likely to be maintained, increase, or decay over time  Tracking form: Region of Waterloo consultations occurred prior to policy change in seven Ontario communities Interviews: Informants from two communities asserted that engagement with the Waterloo experience directly influenced their adoption of a smoke-free MUDs policy   Comparing dose–response patterns with program theory: Program theory might predict whether increased exposure to an intervention is expected to have a positive, negative, or curvilinear relationship to the intended outcomes  Interviews: Informant assertions that one Ontario community consulted extensively with Waterloo. Notably, that community has the most widespread coverage of housing units included in their smoke-free MUDs policy change   Comparing expert predictions with actual results: For evaluations conducted over a period of time, it is possible to make predictions based on program theory or an emerging theory of wider contributors to outcomes, and then to follow up these predictions over time  N/A   Asking participants: While participants might sometimes have their own reasons for attributing or not attributing changes to an intervention, detailed accounts of their change trajectory can be credible  Interviews: All direct participants consulted in this case study asserted that the Waterloo initiative had influence. There were no participants who did not assert influence   Asking other informants  Interviews: Informants from related associations and exposed communities all asserted Waterloo/Propel influence on their pathway to policy change   Making comparisons across cases: The method of qualitative comparative analysis compares the configurations of different cases to identify the components that produce specific outcomes. Program theory can help to identify the variables that should be included in this analysis  Comparison across cases shows that Waterloo/Propel might be important but not sufficient. Further investigation may reveal that Propel/Waterloo may be necessary but not sufficient for MUDs in Ontario in the present time frame  Counterfactual comparison     Comparing the trajectory before and after the intervention: Time series data can provide a credible estimate of the counterfactual in fairly stable situations  Tracking form: There were no Ontario MUDs prior to the start of the Propel work; however, the Region of Waterloo tracking shows after the Propel work seven communities in Ontario adopted similar smoke-free policies for public housing. Each of these seven communities consulted Waterloo processes/experiences prior to adopting their policies   Thought experiments to develop plausible alternative scenarios: Evidence about policies and procedures and other opportunities in some cases can be used to develop a realistic scenario of the chain of events in the absence of an intervention  Interviews: Informants were asked to speculate what would have happened without the Waterloo/Propel work. No informant suggested that the development of MUDs would have gone the same way without it. Those who estimated suggested that ‘we might not be there yet’ and that the impact was likely extensive, i.e. several years  Critical review     Identifying alternative explanations and seeing if they can be ruled out: Alternative explanations might come from insiders (participants, informants), previous research, or speculation  Interviews: Informants noted that American research revealed an awareness of the issue of second-hand smoke risk in buildings, but that people typically prefer local examples   Identifying and explaining exceptions: Exceptional cases might be successes that were expected to be failures or vice versa. Ideally, an evaluation can explain these, or at least document that they exist, and not lose this information by focusing only on the overall pattern  Tracking form: Among the seven Ontario communities to adopt smoke-free MUDs policies after consultations with Waterloo, one community may be the exception to the rule. This community appears to have changed prior to consultation (albeit they still passed their policy 2 years after Waterloo’s passage)  A third approach to assessing the robustness of the contribution stories was interpreting the stories in relation to pertinent scientific theories. In general, findings supported by well-established theories were considered most robust and plausible. We limited our assessment to the scientific theories that informed our theory-based checklist and examined their relevance to the contribution stories in more depth than the checklist. In the MUDs case, one example is how Kingdon’s (2003) three streams theory enhanced the explanatory power and understanding of the policy change process outside of Waterloo region. According to Kingdon (2003), moving an idea onto or higher up on the policy agenda involves three processes: problems, proposals, and politics. In the MUDs case, the evidence showed that Propel research highlighted air quality as a serious issue in MUDs (i.e. the ‘problem’). Propel research also showed the feasibility of making change, as survey results showed support from residents for smoking bans in MUDs (i.e. the ‘proposal’), and the political climate in this case was an important enabler of change (i.e. the ‘politics’). Theories of innovation diffusion could also help explain the patterns of influence. For example, by showing air quality could be measured, Propel may have assisted in making it more trialable, observable, and tangible, making the evidence for change more concrete. Another use of scientific theories in our analysis was to examine contextual influences. We mapped observed contextual influences onto Pawson’s (2013) four categories of contextual factors. In all cases, several factors within each of the four categories were readily mapped onto multiple steps along the impact pathway. Figure 3 shows a subset of factors in each category for the MUDs case. This mapping was useful to describe the range of contextual influences. It also raised questions for future study about the interplay of contextual factors within and across each of the categories. Figure 3. View largeDownload slide Map of illustrative contextual influences for the smoke-free MUDs nested impact pathway. Figure 3. View largeDownload slide Map of illustrative contextual influences for the smoke-free MUDs nested impact pathway. To arrive at an overall contribution story, we examined the consistency and complementarity of results from the above analyses, both within and across cases. We concluded that Propel’s work to synthesize, report, and socialize research findings had a substantial influence on observed policy changes, and it was sufficient when combined with a set of enabling conditions that spanned aspects of the environment, institutions, interpersonal relations, and individuals. For the MUDs case, the adoption of MUDs smoking ban policies in jurisdictions outside of Waterloo region (nested TOC) was facilitated by the social and legal environment in Ontario. A trend to condemn smoking in others’ space as both rude and hazardous was codified in laws, and strongly accepted as a norm across initiatives and actors in the province. This environment provided opportunity and motivation for change in Ontario that may not have been present elsewhere. Characteristics of the research also supported adoption. The studies provided findings that were local to Ontario and showed tangible evidence of second-hand smoke drift through multiple units. This in turn brought in incumbent legal concerns about protecting residents and the rights of nonsmokers. The evidence was provided and translated by credible organizations (including Propel), and individual champions were engaged effectively to refine, disseminate, and discuss messages from research with their peers. In combination, these factors were sufficient to support policy change in at least seven of eight regions that adopted smoking bans for MUDs over the 5 years following the original research and policy adoption in Waterloo Region. 2.5.3 Iterative approach applying CA Steps 4, 5, and 6 The CA team met regularly to assess the strengths and limitations of the contribution stories, and identify actions to strengthen them. We used a series of questions to guide our review of each case: How credible is the story? Did the initial assumptions occur to allow the next step in the chain to take place? Does the pattern of observed results validate the results logic and TOC? What are the main weaknesses or gaps in the story? What perspectives may be missing? What about other gaps in data? The iterations helped to uncover a small number of additional data sources, including one new informant for each of two cases, social media statistics, and a list of presentations and consultations that were relevant to two cases. The iterations also helped the CA team experiment with different data displays to examine the strengths and limits of the contribution stories. 3. Summary and reflections The primary purpose of this article was to contribute to dialogue and debate about CA methods. We experimented with CA methods (summarized in Table 4), with a goal of discerning those that were ‘good enough’, that is relevant to the objectives of each CA step, sufficiently rigorous to achieve credible results, and feasible to complete with resources that may be realistic for organizations to invest in CA studies. To our knowledge, assessing CA methods through the lens of getting to ‘good enough’ is a unique contribution in itself, and may be applied to CA methods beyond those that we used. Table 4. Summary of CA methods used in this study CA steps  Application  Team composition and approach  Formed a team with internal (to Propel) and an external member. An iterative, problem-solving approach was used  CA Step 1: Select cases for the study  Defined the study purpose and questions.  Selected cases selected that met three criteria: policy adoption achieved, sufficient information available, and transferability  CA Step 2: Develop an initial TOC for each case  Developed an overall logic model for each case using a common format. The format was modeled after the Bennett hierarchy which itself represents a research-based change theory and used proximal markers of research influence  Chose a nested impact pathway for each case that met three criteria: feasible to complete, previously underexplored links in the research impact pathway, and relevant to Propel strategy and decisions  Synthesized selective social science theories that would inform subsequent CA steps. Developed a theory-based checklist using sources nominated by team members as most highly relevant to the nested impact pathways—what helps or hinders research use in policy and the policy change process  Embellished the nested impact pathways into TOCs by making assumptions between links and contextual influences explicit  CA Step 3: Gather evidence on the TOCs  Identified and gathered relevant quantitative and qualitative evidence from existing documentation and interviews with informants. Evidence gathered met three criteria: relevant, sufficient quality, from diverse sources. Snowball sampling and semi-structured interviews were used for informant interviews and informants nominated other sources of information  CA Steps 4, 5, and 6: Assemble, assess, and strengthen the contribution stories  Assembled the contribution stories using a table of evidence organized by the nested impact pathway. Evidence reported represented main themes and consistency of data across data sources  Assessed strengths and limitations of the contribution stories using three methods: analysis of consistency of data in the TOC; analyses of causal claims using congruence, comparisons, and critical review; and interpreting the contribution stories in relation to scientific theories  Used an iterative approach applying CA Steps 4, 5, and 6. Regular CA team meetings were guided by questions to ensure appropriate application of Steps 4, 5, and 6  CA steps  Application  Team composition and approach  Formed a team with internal (to Propel) and an external member. An iterative, problem-solving approach was used  CA Step 1: Select cases for the study  Defined the study purpose and questions.  Selected cases selected that met three criteria: policy adoption achieved, sufficient information available, and transferability  CA Step 2: Develop an initial TOC for each case  Developed an overall logic model for each case using a common format. The format was modeled after the Bennett hierarchy which itself represents a research-based change theory and used proximal markers of research influence  Chose a nested impact pathway for each case that met three criteria: feasible to complete, previously underexplored links in the research impact pathway, and relevant to Propel strategy and decisions  Synthesized selective social science theories that would inform subsequent CA steps. Developed a theory-based checklist using sources nominated by team members as most highly relevant to the nested impact pathways—what helps or hinders research use in policy and the policy change process  Embellished the nested impact pathways into TOCs by making assumptions between links and contextual influences explicit  CA Step 3: Gather evidence on the TOCs  Identified and gathered relevant quantitative and qualitative evidence from existing documentation and interviews with informants. Evidence gathered met three criteria: relevant, sufficient quality, from diverse sources. Snowball sampling and semi-structured interviews were used for informant interviews and informants nominated other sources of information  CA Steps 4, 5, and 6: Assemble, assess, and strengthen the contribution stories  Assembled the contribution stories using a table of evidence organized by the nested impact pathway. Evidence reported represented main themes and consistency of data across data sources  Assessed strengths and limitations of the contribution stories using three methods: analysis of consistency of data in the TOC; analyses of causal claims using congruence, comparisons, and critical review; and interpreting the contribution stories in relation to scientific theories  Used an iterative approach applying CA Steps 4, 5, and 6. Regular CA team meetings were guided by questions to ensure appropriate application of Steps 4, 5, and 6  Some of the methods we used reinforce the value of what may be considered current good practice for CA. One method is the iterative approach we used when applying all CA steps (Mayne 2008). Our regular team meetings to share work in progress, and discuss what was working well and what could be improved facilitated learning and problem-solving across cases. A second method increasingly suggested by others (Delahais and Toulemonde 2012; Sridharan and Nakaima 2012; Mayne 2015) and successfully used by our team was focusing on nested impact pathways within broader TOCs. Consistent with experiences of others, choosing nested TOCs enabled sufficient depth with data collection, analysis, and interpretation. Third, our data collection strategy reinforced the importance of diversity of types of evidence and perspectives. Our mix of quantitative and qualitative information from a variety of sources, and input from informants with diverse vantage points on the nested TOC (e.g. researchers, knowledge users, policy domain experts) added strength to the contribution stories. Fourth, our efforts to engage with relevant scientific theories when applying CA steps respond to recommendations by others (Willis et al. 2017; Patton 2012). The CA methods we used may also provide some innovations. One innovation is composition of our CA team. Team members brought a mix of necessary and complementary strengths. The Propel in-house team members all had in-depth knowledge of Propel’s mandate, activities, and way of working, which was essential to understand and probe perspectives from informants, both internal and external to Propel. Affiliation with Propel also helped to recruit informants. Another strength of CA team members from Propel is that they were not involved directly in the research and knowledge exchange activities for the three case examples. This helped to maintain appropriate objectivity (and perceptions of it) with the collection and interpretation of information for each case. The external team member also contributed to this objectivity. The complementary mix of relevant content expertise (e.g. tobacco control, policy development), evaluation in general and CA in particular, and project management also contributed to a rigorous and feasible approach. A second innovation was related to selection of cases. The criteria we used and selecting multiple cases to conduct simultaneously both contributed to relevance, rigor, and feasibility. The three criteria we applied for case selection (i.e. policy changes, sufficient information, and transferable lessons) resulted in rich cases to answer our primary question of how Propel activities contributed to changes in tobacco control policies. Although not an explicit criterion, the common focus on tobacco control policy facilitated problem-solving across cases, and also allowed for shared and efficient data collection (e.g. informants who could provide insights on two of the cases in a single interview). Another promising innovation may be the four tasks we used to complete CA Step 2—developing a TOC. Our approach to developing overall logic models (Task 1) enhanced both relevance and rigor in our study. Most helpful were using the Bennett hierarchy for results logic, which allowed us to emphasize stakeholder engagement and participation, and including proximal markers of research influence (see Pasanen and Shaxson 2016) in the impact pathways, which helped to reveal the inherent complexity in how research may influence policy change (Nutley et al. 2007; Boaz et al. 2009). Both of these choices were instrumental in guiding implementation of subsequent tasks in CA Step 2, as well as subsequent CA steps. With respect to Task 2—choosing a nested impact pathway—relevance and feasibility were strengthened by the criteria we used. Feasibility was an explicit criterion, and potential contributions to new knowledge and decision-making ensured scientific and practical relevance. Our Task 3 was perhaps the most innovative in this step. Engaging with scientific theories to inform Task 4 and other CA steps enhanced the rigor in our study. At the same time, we needed to bound this task to ensure it was practical to complete. The subset of theories we chose was very useful in adding theory-based assumptions and contextual influences to the nested impact pathways. Additional value was realized in later CA steps, as described below. Finally, our team explored innovative methods for assembling evidence and assessing the strength of contribution claims. With respect to assembling evidence, our most promising data display for CA Step 4 was the ‘evidence of impact’ table, adapted from others (Montague and Valentim 2010; Morton 2015). The systematic approach to selecting and presenting evidence in this table helped to identify new data to collect (CA Step 5) and to assess the strength of the contribution stories (CA Step 6). The three methods we used to assess strengths and limitations of the contribution stories—analysis of consistency of data; analyses of causal claims using congruence, comparisons, and critical review; and analysis of the contribution stories in relation to scientific theories—all included some innovation and hold promise for application in other CA studies. The three methods were relevant to the goal of CA Step 6, promoted rigor in our assessments, and were feasible to complete. An innovation in the analysis of consistency of data was explicit criteria for different degrees of consistency (e.g. triangulation of evidence from multiple sources and perspectives as most consistent, and data that were contradictory and/or had significant limitations or gaps as least consistent). Perhaps one of the most promising innovations, especially for highly context-sensitive phenomenon such as policy change, may be our mapping of contextual influences along the impact pathway using Pawson’s (2013) categories of contextual factors. This was an initial step toward accounting for influencing factors and identifying possible alternative explanations, both identified as important areas for CA elaborations (Dybdal et al. 2011; Lemire et al. 2012). 3.1 Limitations An overarching limitation was use of retrospective cases. Retrospective cases meant that data sources included historical recollections and written documentation that was not necessarily aligned with the purposes of the study. Understandably, this resulted in incomplete and potentially inaccurate data, and precluded any comparative analysis over time except as it was (inconsistently) historically recorded or recalled. A prospective approach—laying out anticipated theories of change and monitoring progress against them as they unfold—would allow more rigorous and complete explorations and explanations of influence over time (Morton 2015). This would involve ongoing tracking in areas such as reach, engagement, and early reaction of key groups involved in particular impact pathways. Relevant policy domains could also be tracked over a longer period using a prospective approach. This is especially important in population and public health to allow sufficient time for impacts to manifest (Biggs et al. 2014). 3.2 Possible uses of our findings At the outset, we identified three audiences for the CA applications reported in this article. Those with an interest in how research can be used effectively to influence policy processes and outcomes may benefit from insights related to use of theory. One insight is the potential value of using both program (TOC) and scientific theories for planning how research may be used to influence policy change. Our experience also reinforces the importance of integrating theories of engagement, policy change, and spread of innovations for examining the impact of applied health research on the policy process and outcomes. Those with an interest in evaluating research impacts on policy change may benefit from applying and extending the CA methods we used. Those with an interest in applying CA to other policy domains (e.g. beyond health and policies other than government regulations), and different stages of the policy process (e.g. policy implementation, enforcement) may also apply our CA methods, and explore their transferability. 4. Conclusion CA is a promising response to the challenge of understanding and improving the influence of research on policy, at least in the public health domain and for regulatory approaches to tobacco control. By applying CA retrospectively to three cases of tobacco control policy influenced by a Canadian research center (Propel), we offer methods that are sufficiently relevant, rigorous, and feasible—‘good enough’. Getting to ‘good enough’ required careful selection of nested theories of change; strategic use of social science theories, and quantitative and qualitative data from diverse sources; and complementary methods to assemble and analyze evidence for testing the nested theories of change. CA applications will be strengthened further by prospective data collection. Our experience may inform efforts to influence policy with research, to evaluate research impacts on policy using CA, and to apply CA more broadly. Acknowledgements The authors gratefully acknowledge the participation of all informants in the three policy case examples. The authors also acknowledge contributions made by Alyssa Zarnke to developmental work related to the FT case. Funding This work was supported by the Canadian Cancer Society [2011-701019]. Contributions by Cameron D. Willis were supported by The Australian Prevention Partnership Centre through the National Health and Medical Research Council Partnership Centre grant scheme [GNT9100001] with the Australian Government Department of Health, the New South Wales Ministry of Health, Australian Capital Territory Health, Hospitals Contribution Fund (HCF), and the HCF Research Foundation. Cameron D. Willis is supported by a National Health and Medical Research Council Sidney Sax Fellowship [1013165]. References Bennett C., Rockwell K. ( 1995) Targeting Outcomes of Programs (TOP): An Integrated Approach to Planning and Evaluation . Lincoln, NE: University of Nebraska. Biggs J. S. et al.   ( 2014) ‘ A Practical Example of Contribution Analysis to a Public Health Intervention’, Evaluation , 20 / 2: 214– 29. Google Scholar CrossRef Search ADS   Boaz A., Fitzpatrick S., Shaw B. ( 2009) ‘ Assessing the Impact of Research on Policy: A Literature Review’, Science and Public Policy , 36/ 4: 255– 70. Google Scholar CrossRef Search ADS   Buckley A. P. ( 2016) ‘ Using Contribution Analysis to Evaluate Small & Medium Enterprise Support Policy’, Evaluation , 22/ 2: 129– 48. Google Scholar CrossRef Search ADS   Cancer Research UK ( 2015) Worldwide Cancer Incidence Statistics . London: Cancer Research UK. Corbin J., Strauss A. ( 2008) Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 3rd ed. Thousand Oaks, CA: Sage Publications. Google Scholar CrossRef Search ADS   Dauphinee W. D. ( 2015) ‘ The Role of Theory-Based Outcome Frameworks in Program Evaluation: Considering the Case of Contribution Analysis’, Medical Teacher , 37/ 11: 979– 82. Google Scholar CrossRef Search ADS PubMed  Delahais T., Toulemonde J. ( 2012) ‘ Applying Contribution Analysis: Lessons from Five Years of Practice’, Evaluation , 18/ 3: 281– 93. Google Scholar CrossRef Search ADS   Dybdal L., Steffen B. N., Lemire S. ( 2011) ′ Contribution Analysis Applied: Reflections on Scope and Methodology′, The Canadian Journal of Program Evaluation , 25/ 2: 29– 57. Funnell S. C., Rogers P. J. ( 2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models . San Francisco: Jossey-Bass (Wiley). Greenhalgh T. et al.   ( 2016) ‘ Research Impact: A Narrative Review’, BMC Medicine , 14: Guthrie S., et al.   ( 2013) Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Santa Monica, CA: RAND Corporation. Hanney S., Packwood T., Buxton M. ( 2000) ‘ Evaluating the benefits from health research and development centres’, Evaluation , 6/ 2: 137– 60. Google Scholar CrossRef Search ADS   Kingdon J. W. ( 2003) Agendas, Alternatives, and Public Places . New York: Longman. Kok M. O., Schuit A. J. ( 2012) ‘ Contribution Mapping: A Method for Mapping the Contribution of Research to Enhance Its Impact’, Health Research Policy and Systems , 10: 21. Google Scholar CrossRef Search ADS PubMed  Leeuw F. L. ( 2012) ‘ Linking Theory-Based Evaluation and Contribution Analysis: Three Problems and a Few Solutions’, Evaluation , 18/ 3: 348– 63. Google Scholar CrossRef Search ADS   Lemire S. T., Nielsen S. B., Dybdal L. ( 2012) ‘ Making Contribution Analysis Work: A Practical Framework for Handling Influencing Factors and Alternative Explanations’, Evaluation , 18/ 3: 294– 309. Google Scholar CrossRef Search ADS   LSE Public Policy Group ( 2011) Maximizing the Impacts of Your Research: A Handbook for Social Scientists. London: LSE Public Policy Group. Mark M., Henry G. ( 2004) ‘ The Mechanisms and Outcomes of Evaluation Influence', Evaluation ’, 10/ 1: 35– 57. Google Scholar CrossRef Search ADS   Mayne J. ( 2008) ‘ Contribution Analysis: An Approach to Exploring Cause and Effect’, The Institutional Learning and Change (ILAC) Initiative . ILAC Brief 16. Mayne J. ( 2012) ′ Contribution Analysis: Coming of Age?′, Evaluation , 18/ 3: 270– 80. Google Scholar CrossRef Search ADS   Mayne J. ( 2015) ′ Useful Theory of Change Models′, Canadian Journal of Program Evaluation , 30/ 2: 119– 42. Google Scholar CrossRef Search ADS   Mertens D. M., Hesse-Biber S. ( 2013) ‘ Mixed Methods and Credibility of Evidence in Evaluation’, New Directions for Evaluation , 2013/ 138: 5– 13. Google Scholar CrossRef Search ADS   Milat A. J., Bauman A. E., Redman S. ( 2015) ‘ A Narrative Review of Research Impact Assessment Models and Methods’, Health Research Policy and Systems , 13: 7. Google Scholar CrossRef Search ADS PubMed  Montague S. ( 2015) ′The Need to Build Reach Into Results Logic, Theories of Change and Performance Frameworks′, in CES 2015 Annual Conference, Montreal. Montague S., Valentim R. ( 2010) ‘ Evaluation of RT&D: From ‘Prescriptions for Justifying’ to ‘User-Oriented Guidance for Learning’’, Research Evaluation , 19/ 4: 251– 61. Google Scholar CrossRef Search ADS   Morton S. ( 2015) ‘ Research Impact Assessment: A ′Contributions′ Approach’, Research Evaluation , 24/ 4: 405– 19. Google Scholar CrossRef Search ADS   Newcomer K., Brass C. T. ( 2016) ‘ Forging a Strategic and Comprehensive Approach to Evaluation Within Public and Nonprofit Organizations. Integrating Measurement and Analytics Within Evaluation’, American Journal of Evaluation , 37/ 1: 80– 99. Google Scholar CrossRef Search ADS   Nutley S. M., Walter I., Davies H. T. O. ( 2007) How Research can Inform Public Services . Bristol, UK: Policy Press, 376. Google Scholar CrossRef Search ADS   Pasanen T., Shaxson L. ( 2016) How to Design a Monitoring and Evaluation Framework for a Policy Research Project [online text], Overseas Development Institute. Patton M. Q. ( 2012) ‘ A Utilization-Focused Approach to Contribution Analysis’, Evaluation , 18/ 3: 364– 77. Google Scholar CrossRef Search ADS   Pawson R. ( 2013) The Science of Evaluation: A Realist Manifesto . Thousand Oaks, CA: Sage. Google Scholar CrossRef Search ADS   Penfield T. et al.   ( 2014) ‘ Assessment, Evaluations, and Definitions of Research Impact: A Review’, Research Evaluation , 23/ 1: 21– 32. Google Scholar CrossRef Search ADS   Rogers E. M. ( 1995) Diffusion of Innovations , 4th edn. New York: The Free Press. Rogers P. ( 2008) ‘ Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions’, Evaluation , 14/ 1: 29– 48. Google Scholar CrossRef Search ADS   Sridharan S., Nakaima A. ( 2012) ‘ Towards an Evidence Base of Theory-Driven Evaluations: Some Questions for Proponents of Theory-Driven Evaluation’, Evaluation , 18/ 3: 378– 95. Google Scholar CrossRef Search ADS   Stachowiak S. ( 2013) Pathways for Change: 10 Theories to Inform Advocacy and Policy Change Efforts. Seattle: ORS Impact; Washington, DC: The Center for Evaluation Innovation. Willis C. D., Riley B., Stockton L., Viehbeck S., Wutzke S., Frank J. ( 2017) ‘ Evaluating the impact of applied prevention research centres: results from a modified Delphi approach’, Research Evaluation , 26/ 2: 78– 90. Google Scholar CrossRef Search ADS   Wimbush E., Montague S., Mulherin T. ( 2012) ‘ Applications of Contribution Analysis to Outcome Planning and Impact Evaluation’, Evaluation , 18/ 3: 310– 29. Google Scholar CrossRef Search ADS   © The Author 2017. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

Journal

Research EvaluationOxford University Press

Published: Jan 1, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off