Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

A time-responsive tool for informing policy making: rapid realist review

A time-responsive tool for informing policy making: rapid realist review Background: A realist synthesis attempts to provide policy makers with a transferable theory that suggests a certain program is more or less likely to work in certain respects, for particular subjects, in specific kinds of situations. Yet realist reviews can require considerable and sustained investment over time, which does not always suit the time-sensitive demands of many policy decisions. ‘Rapid Realist Review’ methodology (RRR) has been developed as a tool for applying a realist approach to a knowledge synthesis process in order to produce a product that is useful to policy makers in responding to time-sensitive and/or emerging issues, while preserving the core elements of realist methodology. Methods: Using examples from completed RRRs, we describe key features of the RRR methodology, the resources required, and the strengths and limitations of the process. All aspects of an RRR are guided by both a local reference group, and a group of content experts. Involvement of knowledge users and external experts ensures both the usability of the review products, as well as their links to current practice. Results: RRRs have proven useful in providing evidence for and making explicit what is known on a given topic, as well as articulating where knowledge gaps may exist. From the RRRs completed to date, findings broadly adhere to four (often overlapping) classifications: guiding rules for policy-making; knowledge quantification (i.e., the amount of literature available that identifies context, mechanisms, and outcomes for a given topic); understanding tensions/ paradoxes in the evidence base; and, reinforcing or refuting beliefs and decisions taken. Conclusions: ‘Traditional’ realist reviews and RRRs have some key differences, which allow policy makers to apply each type of methodology strategically to maximize its utility within a particular local constellation of history, goals, resources, politics and environment. In particular, the RRR methodology is explicitly designed to engage knowledge users and review stakeholders to define the research questions, and to streamline the review process. In addition, results are presented with a focus on context-specific explanations for what works within a particular set of parameters rather than producing explanations that are potentially transferrable across contexts and populations. For policy makers faced with making difficult decisions in short time frames for which there is sufficient (if limited) published/research and practice-based evidence available, RRR provides a practical, outcomes-focused knowledge synthesis method. Keywords: Realist review, Rapid review, Evidence synthesis, Health policy * Correspondence: [email protected] Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute, The University of British Columbia, Research Pavilion, 7th Floor, 828 West 10th Avenue, Vancouver, BC V5Z 1M9, Canada InSource Research Group, 6975 Marine Drive, West Vancouver, BC V7W 2T4, Canada Full list of author information is available at the end of the article © 2013 Saul et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Saul et al. Implementation Science 2013, 8:103 Page 2 of 15 http://www.implementationscience.com/content/8/1/103 Background a high level of training and experience is required, which Efforts to improve the connections between research may not be found routinely in government or policy devel- evidence and policy decisions have resulted in a number opment agencies, academic institutions, or community- of sophisticated synthesis tools [1]. Pawson has broadly based organizations. Perhaps most relevant to our current classified evidence synthesis approaches as being meta- discussion, realist reviews as outlined by Pawson et al. analytical (re-analysing data from numerous studies to [3,8] can require considerable and sustained investment arrive at broadly generalizable findings) or narrative over time, which does not always suit the time-sensitive (using text-based data extractions to compare study demands of many policy decisions. In addition, due to findings as a way for understanding why programs or in- their expansive and exploratory nature, realist reviews can terventions have certain effects), both of which are com- often suffer from ‘scope creep’. monly employed for informing policy decisions [2]. ‘Rapid reviews’ have emerged in response to the incom- While Pawson notes the relative advantages of meta- patibility between information needs of policy makers and analytical and narrative approaches, each technique also the time requirements to complete systematic reviews. has key limitations: meta-analyses fail to provide under- Rapid reviews provide a way to generate similar types of standing of how programs work, while narrative reviews knowledge synthesis as more comprehensive systematic give little insight into how a program may operate in reviews do, but in a much shorter time period. While some different settings or under different circumstances [3,4]. have questioned the validity of rapid reviews [9,10], there Limited generalizability of study results for a given remains a need to achieve a balance between comprehen- context has been addressed as a common concern in siveness and timeliness for many policy-relevant decisions evidence-informed public health policy [5,6]. Alterna- [11]. Therefore, understanding how this realist approach tively, we have realist reviews, which are an approach to may be applied in cases where there is limited time and re- synthesising evidence first promoted by Pawson [4]. sources to generate a realist-based product that can in- Pawson notes that the underlying principles of realist corporate research, theory, and practice knowledge and approaches are the links between interventions (I), con- thus meet the demands of real-time policy developers/ texts (C), mechanisms (M), and outcomes (O) [4]. In evaluators, is a valuable contribution to both evidence- contrast to specific activities included as part of an inter- informed policy-making and the realist review literature. vention or program, mechanisms (M) are thought to be ‘Rapid Realist Review’ methodology (RRR) has been de- the responses by people that are triggered by changes in veloped by members of our group as a tool for applying a context (C) [7]. Traditional realist syntheses use these realist approach to a knowledge synthesis process and C-M combinations to generate transferable ‘program producing a product that is useful to policy makers in theories’ that suggest that certain interventions are more responding to time-sensitive and/or emerging issues or less likely to work in certain ways, for certain people, where there is limited time and resources. Where relevant, in certain situations [3,4]. In other words, how does a we compare the RRR methodology to realist reviews gen- change in context generate a particular mechanism that erally, or ‘traditional’ realist reviews. While there is no set in turn produces specific outcomes? definition for what constitutes a realist review other than In developing these program theories, realist reviews its use of realist logic and constructs, we present ‘trad- assume a broad inclusion of evidence, both quantitative itional’ realist reviews as those that typically engage in a and qualitative, which is used to develop an understand- much longer exploration of the literature and ‘testing’ of ing of ‘what works, for whom, in what contexts, to what theories, as well as those that present results within a extent, and most importantly how and why [2,3]?’ framework of theory development. While others have Therefore, instead of offering prescriptive advice on begun to describe methodologies for short-term evidence what the ‘best buys’ might be for policy makers, a realist synthesis projects [12], the RRR methodology described synthesis attempts to provide a transferable theory that here is intended to incorporate the theory specification of suggests that a certain program is more or less likely to a realist review [8] and the boundary clarification aim of a work in these respects, for these subjects, in these kinds scoping review [13]. (See [11] for an example of an earlier of situations [2,3]. review). This is also consistent with the recently published However, realist review methods are not without their RAMESES (Realist And MEta-narrative Evidence Synthe- difficulties. The iterative, flexible nature of realist reviews ses: Evolving Standards) publication standards for realist does not align well with protocol-driven, standardized pro- syntheses, which note that realist reviews need to be fo- cesses common to established systematic review methods cused based on the time and resources provided as well as [8]. As results from a realist review are context dependent, the questions that need to be answered [14]. the generalizability of findings to other settings will depend In our experience, policy makers have been more in- on the operation of similar mechanisms to generate out- terested in knowledge syntheses that highlight possible comes of interest. Moreover, in completing a realist review, interventions (I) that could be implemented within a Saul et al. Implementation Science 2013, 8:103 Page 3 of 15 http://www.implementationscience.com/content/8/1/103 specific context (C) that in turn interact with various incorporated it within the RRR method explicitly as a mechanisms (M) and produce outcomes (O) of interest. means to streamline the review process. Other variations As a result, the emphasis of our reviews has changed over that are distinct from more traditional realist reviews are time from one focused on producing transferable theories, noted below. to one focused on identifying groups of interventions re- lated to outcomes of interest for policy makers. Our modi- 1. Development of the project scope: clarifying with fied review methodology ‘works backwards’ from the the knowledge users the content area of interest for desired outcome (e.g., large scale transformation of health the review. As with any type of realist review, this systems, sustainable cultural change, etc.), to ‘families of step is critical in ensuring a feasible review process, interventions’ (I) that can be implemented to produce regardless of the desired timeline. those outcomes, supported by a theoretical understanding 2. Development of specific research questions: once of the contexts (C) within, and mechanisms (M) by which the project scope has been narrowed, discussing the such interventions operate. In doing so, the RRR method- specific questions that knowledge users are most ology focuses less on the development of theory that is interested in answering. Refining these questions to transferable across domains, than on the distillation of ensure that there is enough evidence to be able to theory-driven, contextually relevant interventions that are answer them, at least in part. As with step 1 above, likely to be associated with specific outcomes within a par- this is a critical component of any type of realist ticular set of parameters. review. Here, we describe our experience in developing RRR, 3. Identification of how the findings and the key ingredients in making it work, the policy ques- recommendations will be used: this includes tions to which it is most suited, the strengths and limita- formulating a purpose statement that helps identify tions of the method, and the future directions of how how the findings of the review will be used by the this approach may be optimized in practice. In doing so, target audience. Utilization of review products is a key we provide key examples of RRR projects completed by element in the RRR methodology, and one which our group over the past six years. systematic reviews often do not address. While traditional realist reviews may incorporate this step as Methods well, it is not explicitly included in the RAMESES We provide here brief summaries of key examples of re- publication standards for realist reviews [14]. This views completed to date by this team. Table 1 includes could be considered part of step 2 above, although it the commissioning agencies, title, duration and purpose has allowed us to focus our reviews more clearly with of each review. For this paper, we also considered key the knowledge users in mind throughout the findings and recommendations resulting from each, as searching, extraction, and synthesis process. well as the strengths and challenges that arose while 4. Development of search terms: collaboratively completing the review. We draw from these examples to identify terms likely to be relevant to the project highlight the conceptual strengths and limitations of scope, purpose, and research questions. 5. Identification of articles and documents for inclusion rapid realist review methods and the circumstances under which they offer most value. in the review (both published and grey): begin with a list of documents as identified by knowledge users and content experts. In addition, use the search Methods of realist review As Pawson has described elsewhere [3], key stages and terms to iteratively generate lists of documents that may be included in the review. tasks for a realist review include the following steps: iden- 6. Quality review: narrow the search terms based on tifying the review question, searching for primary studies, quality appraisal, extracting the data, synthesizing the data, the results that are most relevant to the review topic. Simultaneously, poll the knowledge users and and disseminating the findings (see [3], chapter 4). How- external content experts to identify documents ever, while Pawson has stated that realist reviews need to be focused, the methodology he has described has not (published and grey) of key importance for the review. We explicitly acknowledge that a search been explicit about how to focus the review, particularly using the RRR methodology will not be for cases where a policy recommendation is required comprehensive. Polling knowledge users and content quickly. In response to the need for a time- and resource- experts to identify key articles accelerates our search sensitive review process, we have modified and expanded process. This, combined with the validation step on Pawson’s original methodology, including the explicit (#8 below), helps ensure that we are not missing engagement of content experts drawn from stakeholders. significant sections of the literature with our While engagement of experts and stakeholders is a com- abbreviated search. mon strategy for traditional realist reviews, we have Saul et al. Implementation Science 2013, 8:103 Page 4 of 15 http://www.implementationscience.com/content/8/1/103 Table 1 Examples of rapid realist reviews undertaken by our group Commissioning agency Title Purpose statement Duration Public Health Agency of Canada Interventions To Increase Organisational The purpose of this project was to conduct a Six months Capacity To Address Health Literacy (2012) realist review of available literature (published and grey) to summarize what we know and what we don’t know about the development of capacity within organizations to plan, implement, and sustain health literacy interventions. This review did not assess the relative efficacy of health literacy interventions themselves, but rather, attempted to identify what mechanisms and contextual factors improve the capacity of healthcare organizations to address health literacy in the individuals they serve by planning, implementing, evaluating and sustaining interventions. Canadian Institutes of Health Knowledge And Action For System The Saskatchewan Ministry of Health has a Six months Research (Partnering organization - Transformation (KAST): A Systematic Realist mandate to support a significant Saskatchewan Ministry of Health) Review and Evidence Synthesis of the Role of transformation of the provincial health system. Government Policy in Directing Large System Large system transformation refers to Transformation (2010) systematic initiatives to create coordinated change across organizations working toward shared priorities within specified boundaries. The Ministry recognizes that success hinges on a cultural change based on collaboration, a comprehensive innovation strategy for system redesign, and systems integration. Therefore, they seek guidance -- on such considerations as successful models and strategies, partnership principles (including with patients), monitoring and evaluation -- from a systematic review of knowledge on large system transformation. With all of these considerations, there is an emphasis on the role of government. HealthLink BC and QUILTS Tele-health Contributions to Emergency A rapid review of tele-health contributions to Five months Department and Discharge Operations (2009) Emergency Department (ED) and discharge operations was conducted to: 1. Describe the patient population in selected local ED facilities; identify major drivers of ED utilization, throughput and follow-up; and conduct a rapid review of the effectiveness of tele-health strategies for improving ED/ discharge system performance. 2. Scan best practices in other jurisdictions and identify potential opportunities for HealthLink BC to add value for regional health authority ED/discharge systems. 3. Produce a model of system operations and a framework for ways in which HealthLink BC tele-health services might help to improve performance for ED services, and for discharge planning and follow-up. Public Health Agency of Canada Evidence-informed public health policy and This RRR was commissioned to summarize Two months practice through a complexity lens (2007) what is known about systems to support evidence-informed policy and what is not known. The review focused on frameworks most applicable to public health, with an emphasis on chronic disease prevention and healthy living. The review process also initiated an international collaboration with a focus on system development for evidence-informed public health. Saul et al. Implementation Science 2013, 8:103 Page 5 of 15 http://www.implementationscience.com/content/8/1/103 Table 1 Examples of rapid realist reviews undertaken by our group (Continued) HealthLine Services BC Tele-health in Support of Chronic Disease The purpose of this project was to provide Six months Management (2007) HealthLine Services BC (HLSBC) with a review of evidence and best practices for health line support for chronic disease management. Vancouver Coastal Health Population-based Framework for Chronic The overall goal of the project was to identify Three months Disease Self-management Support (2006) the most current view on effective models for self-management and self-management support, and to provide decision makers in the policy and practice community at Vancouver Coastal Health with a practical understanding of evidence-based interventions, within a population-based framework for self- management support. NIH Office of Behavioral and Social Interorganizational partnerships (2006) To provide a RRR of evidence-based principles Two months Science Research and the for effective interorganizational collaboration Canadian Health Services Research and partnership, as the principles relate to two Foundation contexts: the collaborations in which health research funders engage; and the forms and processes of linkage within universities and their community partners to promote interdisciplinary health research and effective knowledge exchange. The review also summarized the existing evidence to identify aspects of interorganizational partnerships (a) where sufficient evidence exists to conduct a full systematic review, and (b) where insufficient evidence exists to articulate the further primary research that is necessary. 7. Extraction of data from the literature: using an recommendations, further knowledge gathering and extraction template, pull out elements from synthesis, or evaluation of knowledge application. documents that can help answer the research Because findings are typically presented to policy questions. Data are extracted using identical makers with a focus on families of interventions that methods to a traditional realist review. Findings are can be implemented within their context, they can analyzed to build a form of realist program theory be easily acted on. Program theories are presented as that addresses the agreed focus and scope of the a way to understand how changes in context may review. As stated earlier, policy makers tend to be interact with mechanisms to produce outcomes of more interested in families of interventions that can interest, rather than as the primary findings. change context (C) in ways that interact with Program theories also help describe potential mechanisms (M) to produce outcomes (O) of unintended consequences resulting from changes in interest, and less interested in program theories. context and their resultant interactions with However, development of these program theories is mechanisms. essential for understanding the C-M interactions that can then help explain why these families of These ten steps are similar to the procedure for a interventions will or won’t work in the policy Cochrane or other systematic review, although they fun- maker’s setting. damentally differ due to their iterative nature (i.e., steps 8. Validation of findings with content experts: once 1 through 7 may be revisited iteratively throughout the program theories have been generated, they are RRR process) and the rapidity with which they are reviewed by content experts who have direct conducted. The primary differences involve the realist experience in the field to ensure that they represent philosophical approach to how the extraction, analysis the learnings of practitioners, and to fill any gaps and synthesis are completed (focusing on how context that may have been left by the published literature. interacts with mechanisms to produce outcomes), and 9. Synthesis of the findings in a final report. The report the involvement of knowledge users in shaping the final is formatted in a way intended to meet the needs of product. the knowledge users, based on the results of step 3 It is increasingly a necessity to engage knowledge users above, and the findings produced by steps 7 and 8. in order to receive funding to conduct reviews of all 10.Dissemination of results: working with the types. This fits well with the RRR design, and also con- knowledge users to apply the findings through policy stitutes one of its critical strengths. In particular, the Saul et al. Implementation Science 2013, 8:103 Page 6 of 15 http://www.implementationscience.com/content/8/1/103 involvement of a local reference group and expert panel scratch, identification and engagement of a group of con- facilitate both more efficient identification of key mate- tent experts can be enormously time consuming. How- rials to include in the review, and sufficiently robust ever, we have been able to circumvent the process of findings. Involvement of knowledge users and external identification, introduction, invitation and engagement experts also increases both the usability of the review due to our ability to draw on first- and second-degree net- products, as well as their link to current practice. Our work connections. engagement of a local reference group and an expert The initial search places a special emphasis on grey lit- panel is consistent with the recommendations put forth erature that may not be easily accessible through trad- by Lavis et al. to create a ‘dialogue that allows the data itional search methods, especially for cases in which the and research evidence … to inform and be considered published literature on the selected topic may be of lim- alongside the views, experiences, and tacit knowledge of ited relevance to the specific research question at hand. those who will be involved in, or affected by, future deci- Inclusion and exclusion criteria are developed to assist sions about the health system problem’ [15]. with selection of documents to review further. Based on At all stages of the process, a local reference group is these criteria, a screening tool is developed that is used engaged to ensure that the project will produce results to screen abstracts that are identified using the initial that will be relevant for the context in which they will search terms. As the search progresses, search terms and be used. The reference group typically includes repre- inclusion/exclusion criteria are modified iteratively with sentatives of the funding organization, as well as know- the goal of identifying the documents most relevant to ledge users (the target audience for the findings of the the review topic without missing many that might be review). As Keown et al. have shown, the potential bene- important to include. The expert panel is asked to con- fits of including stakeholders in the process of a review tribute a list of the most relevant articles at the start of include increased relevance, clarity and awareness of re- the search to streamline the process. The importance of view findings [16]. Van de Ven [17,18] provides a similar the expert panel’s and reference group’s help in the view for coproduction that he calls ‘engaged scholarship’. search process cannot be underestimated with respect to Given the explicit intention of the RRR methodology to conducting a rapid review. provide results that can be used by policy makers, these Once an initial list of documents has been identified, benefits become critical to the success of a project. pilot extractions are conducted with all members of the In addition to the local reference group, the RRR meth- review and extraction team. The extraction template in- odology identifies an expert panel made up of researchers cludes fields such as ‘theoretical foundation or concep- and practitioners, actively engaged in conducting work in tual framework’, ‘facilitators/barriers to success of the the content area for the review, who are in the process of project’ (e.g., contextual changes that helped or hindered negotiating the interplay between research, practice and the triggering of mechanisms to produce both intended policy. Engagement of an expert panel provides several and unintended outcomes), and ‘interactions between benefits, including focusing the review scope; streamlining context and mechanisms’. Template fields are modified the search process; filling gaps in the initial problem defin- and tailored to meet the needs of each individual review ition, search process or synthesis of findings; and ensuring project. A series of calibration exercises consist of com- appropriate interpretation of results. paring the extractions of the various team members for While these experts are often scattered internationally, level of detail, identification of relevant themes, and in- several factors have allowed us to rapidly convene such dividual style for presentation of findings. Between three panels within a matter of weeks from project initiation. and five articles are typically reviewed in this fashion. First, knowledge users often have connections to one or Further extractions are conducted by individual team more content experts for the topic, given their interest members. Additional consultations are conducted as and involvement with the topic. During initial discussions needed throughout the review process. As documents of project scope and research questions, collectively we are extracted, forward and backward citation searches identify experts who already have connections to the pro- are conducted on key articles. Additional documents are ject team or knowledge users. Initial invitations are ex- added to the review list throughout the process. Because tended to this small group who are fit for purpose for a the timeline for each rapid review is typically tight (six given review. Additional targeted invitations are extended months or less), there is a limited window of opportun- to those publishing within the domain of interest. Interest ity to add documents to the review. This emphasizes the and engagement can often be enhanced by further invi- importance of accelerating the review process with sev- tations coming from, or mentioning, experts who have eral key documents that can be used for forward and already agreed to participate. Finally, the time commit- backward searching. ment required by expert panellists is deliberately mini- After the initial round of extractions is conducted, the mized, and highlighted in the invitation. If starting from review team summarizes key themes and findings, Saul et al. Implementation Science 2013, 8:103 Page 7 of 15 http://www.implementationscience.com/content/8/1/103 focusing on contextual factors and mechanisms that im- positivist, linear, cause-and-effect perspective, but rather pact outcomes, and how context and mechanisms interact. from the perspective that mechanisms for change will al- Typically, a matrix is created listing contextual factors and ways be context-dependent. It is this grounding in the mechanisms. Findings are grouped by similar contextual realist philosophical tradition that distinguishes realist changes and how they trigger mechanisms to produce out- reviews from other types of systematic reviews. At the comes (demi-regularities). These groupings are regularly same time, it is essential that the review team have a shared and discussed within the review team to ensure clear sense of the scope and purpose of the review to en- validity and consistency in the inferences made, although sure maximum utility and application of the results for due to the time restrictions, this typically occurs twice or each project’s unique audience. Familiarity with the con- three times during the review. Findings are reviewed sim- text in which the results will be used (e.g., policy devel- ultaneously with the local reference group. Based on the opment) is also helpful. preliminary findings, the project questions, inclusion and Team composition differs for an RRR as compared to exclusion criteria, and the fields for the extraction tem- a traditional realist review in that the project manager plate are adjusted as needed. Following this process, the must be disciplined and able to keep a large complex extraction process is completed, and a draft of key findings process moving along according to schedule. The local is presented to both the local reference group and expert reference group and expert panel must be prepared to panel. Feedback is obtained to ensure relevance to the provide feedback in a timely fashion, as this can often be local context of knowledge users, consistency with the a limiting factor in how quickly a review can be com- body of published and grey literature, and consistency pleted. And finally, the synthesis lead must be able to re- with the current practice and tacit knowledge of the con- view large quantities of information efficiently, distil it tent experts. Feedback is used to expand on the findings into easily digestible components for review by the refer- and tailor the synthesis for the final report. ence group and expert panel, and quickly incorporate Team members needed to complete a Rapid Realist feedback while keeping the intended uses of the final Review include: product in mind. The reference group (knowledge users) must also be clear about their priorities, which can 1. Project manager, responsible for preparing internal help the review team make decisions about the review project documents, coordinating the dialogue and process, particularly as the project deadline approaches. managing a pre-determined set of requests for the Panel involvement varies in intensity over time. Clarity reference group and expert panel (providing of purpose and timelines, and a willingness of panellists feedback at each stage of the process from question to respond within the allocated time frame, are essential development to review of the final report), components of a project as well. consolidating feedback, maintaining the timeline, budget and other duties; Results 2. Local Reference group (including client Table 1 provides details of the RRRs completed by this representatives) and expert panel (ideally four to six team to date. Agencies commissioning RRRs have in- individuals for each group); cluded Federal Government Agencies (e.g., Public Health 3. Librarian (or information specialist) to lead on Agency of Canada, U.S. National Institutes of Health), document searches; Provincial Health Ministries (e.g. Saskatchewan Ministry 4. Review team (two to four individuals who screen of Health) and associated initiatives (e.g., Healthlink BC), abstracts, read selected documents, and perform regional health authorities (e.g., Vancouver Coastal Health extractions); Authority), and key research funding bodies (e.g., Canadian 5. Synthesis lead to oversee the review process and Institutes of Health Research). In a number of instances, play a main role in synthesizing information; reviews have been funded by research granting agencies 6. Academic or research lead. for work to be completed in partnership with identified collaborators. The composition of the review team is critical for the There is considerable variation in the topical focus of RRR methodology. All team members must be well- each RRR included in Table 1. Reviews often have been versed in the realist philosophy that serves as the theor- commissioned to synthesize information on complex, sys- etical underpinning for realist reviews. In addition, the tem oriented issues, such as large system transformation, academic or research lead and synthesis lead must not improving organizational competencies for addressing only be familiar with realism and realist approaches to health literacy needs, enhancing interorganizational part- synthesis reviews, but must also have participated in a nerships, and improving evidence-informed health policy realist review process previously. It is critical that the re- as well as bettering the people-centredness of health au- view team members approach the project not from a thorities. In addition, a number of reviews have focused Saul et al. Implementation Science 2013, 8:103 Page 8 of 15 http://www.implementationscience.com/content/8/1/103 on specific clinical specialties, domains or services, such confirming the superiority of multi-faceted chronic disease as tele-health interventions, emergency department dis- prevention programs compared to non-multi-faceted ap- charge practices, and optimal care models for chronic dis- proaches within a specific context for particular popula- ease management programs. tions, or identifying that large scale changes in Canadian As outlined in Table 1, RRRs may be completed for a health systems at this moment in time require engaged variety of purposes. Two broad purpose domains appear to physician and patient groups. Because they highlight I-C-M emerge, with RRRs used either to inform the development interactions, they also make apparent situations when simi- of practical solutions to problems faced in care delivery, or lar types of interventions would not result in their intended to develop a more conceptual understanding of how to outcomes due to contextual differences triggering alternate organize ideas or frameworks for refining high level stra- mechanisms. RRRs make explicit the amount of literature tegicthinking(Table1). Forthe former,RRRshavebeen readily available for review, and in the case of a limited evi- used to synthesize best practices for specific service dence base, can draw on the input of the expert panel in provisions, such as evidence-based tele-health initiatives understanding the absence of literature and subsequently or an inventory of self-management interventions for developing directions to take in sourcing other relevant lit- chronic disease patients. More conceptual applications have erature for the review. In addition to understanding when produced ‘simple rules’ for guiding large system transform- little is known on a given topic, RRRs also highlight where ation, syntheses of frameworks for supporting evidence- tensions or uncertainties exist in an evidence base, such as informed policy-making and strategies for fostering what the precise characteristics of an effective network organizational partnerships. Methodologically, RRRs have might be for a given purpose within a specific geographic, offered decision makers a tool for merging sources of evi- political and temporal context, or the differential effective- dence including published literature, grey literature and ness of tele-health methods for chronic disease self- expert opinion, and as such, have provided knowledge management for populations within varying contexts. users with the ability to make sense of how and why they The RRRs cited in this analysis have a number of might expect certain outcomes to occur, and the role of strengths. Engaged members of both the expert panel and context in these outcomes. Further, they have helped reference group ensure rigorous methods by providing an knowledge users determine if sufficient substrate exists for extended body of expertise and experience, and validating other structured review exercises. In doing so, RRRs have findings and program theories in an iterative way. In proven useful in providing evidence for and making expli- addition, the participation of the reference group increases cit what is known on a given topic, as well as articulating the local utility of review findings. Moreover, the results where knowledge gaps may exist. speak to the value of diverse data collection methods, From the RRRs completed to date, findings broadly ad- building on the literature through methods such as sur- here to four (often overlapping) classifications: guiding veys and focus groups. In addition, reference groups tend rules for policy making (i.e., evolving patterns of decision- to include (or provide links to) key decision makers in making that can be adapted based on new knowledge commissioning agencies, leading to closer connections be- or contextual factors); knowledge quantification (i.e.,the tween knowledge producers and users. As such, the RRRs amount of literature available that identifies context, mech- completed to date have generally performed well as tools anisms and outcomes for a given topic); understanding for responding to time-sensitive policy issues. tensions/paradoxes in the evidence base, as when similar As with any review process, there are challenges to the changes in context trigger similar mechanisms and produce RRR process. Ensuring that the scope of the review re- divergent outcomes, or when changes in context trigger mains contained and the questions definable can be espe- different mechanisms to produce the same or different out- cially important and difficult in the RRR process. Like all comes; and, reinforcing or refuting beliefs and decisions reviews, difficulties may emerge when limited published or grey literature exists, where the quality of included stud- taken. See Table 2 for case studies that include detail on specific findings for two reviews. RRR provides a tool for ies is poor, or where the findings of studies are heavily developing rules for guiding decision-making, such as the contextualised to particular settings or times. Forming the membership of the reference group and ensuring sustained advantages of investing in both top-down and distributed leadership, as well as on-going performance measurement involvement are challenges for all reviews, particularly in in large system transformations (e.g.,Knowledge AndAc- rapidly changing political environments where member- ship of the reference group may change during the course tion For System Transformation (KAST) review), or the need for innovative inter-organizational investments for of a given project. improving the links between knowledge creation and adop- tion. While these guiding rules can inform the deve- Discussion lopment of strategic directions, they can also be used as The RRR method outlined here and a more traditional tools to confirm or dispel existing beliefs; for example, realist review share numerous similarities (see Table 3). Saul et al. Implementation Science 2013, 8:103 Page 9 of 15 http://www.implementationscience.com/content/8/1/103 Table 2 Case studies Project title: Knowledge And Action For System Transformation (KAST): A Interorganizational Partnerships Systematic Realist Review and Evidence Synthesis of the Role of Government Policy in Directing Large System Transformation Partner: Saskatchewan Ministry of Health National Institutes of Health - Office of Behavioral and Social Science Research and the Canadian Health Services Research Foundation Funder: Canadian Institutes of Health Research Same as partner Length of Six months Two months time: Published Best A, Greenhalgh T, Lewis S, Saul J, Carroll S, Bitz J 2012. Large Riley BL, Best A. (in press). Stakeholders, organizational article: system transformation in health care: A realist systematic review and partnerships, and coalitions. In S. Kahan, A. Gielen, P. Fagan, & evaluation of its usefulness in a policy context. The Milbank L.W. Green (eds). Health Behavior Change in Populations: The Quarterly, 90(3): 421–456. State of the Evidence and Roles for Key Stakeholders. Baltimore, MD: Johns Hopkins University Press. Purpose: The Saskatchewan Ministry of Health has a mandate to support a To provide a RRR of evidence-based principles for effective significant transformation of the provincial health system. Large interorganizational collaboration and partnership, as the system transformation refers to systematic initiatives to create principles relate to two contexts: the collaborations in which coordinated change across organizations working toward shared health research funders engage; and the forms and processes priorities within specified boundaries. The Ministry recognizes that of linkage within universities and their community partners to success hinges on a cultural change based on collaboration, a promote interdisciplinary health research and effective comprehensive innovation strategy for system redesign, and knowledge exchange. The review also summarized the systems integration. Therefore, they seek guidance -- on such existing evidence to identify aspects of interorganizational considerations as successful models and strategies, partnership partnerships (a) where sufficient evidence exists to conduct a principles (including with patients), monitoring and evaluation -- full systematic review, and (b) where insufficient evidence from a systematic review of knowledge on large system exists to articulate the further primary research that is transformation. With all of these considerations, there is an emphasis necessary. on the role of government. Questions 1. What are the key mechanisms or social processes that influence 1. What are the evidence, experiential and theory-based asked in the or drive successful large systems transformation in the health care critical success factors for increased interorganizational review: sector? collaboration? 2. What are the contextual factors that have the most impact 2. What is the nature of the evidence, and what is known (positive or negative) on large systems transformation efforts in the about external validity and generalizability? healthcare sector? 3. Are there identifiable ‘transition’ points in large systems 3. What are the best indicators to monitor collaboration and transformation efforts? If so, how do the key mechanisms and outcomes a) within the research funder context, b) within the contexts interact to produce these changes? university/institutional (researcher performer) context, and c) within the context of the users of research knowledge (research user)? 4. What is the role of government or government-like entities (these 4. What are the strategies most likely to generate buy-in of need definition or at least examples) in large system transformation key influentials in each context for strategic plans and new efforts? initiatives? 5. What does the literature say about the challenges or constraints most likely to affect success in each context? 6. What does the literature say about stages of the development of collaborations and the most effective tactics at each stage? Resulting (In the full report, each has an associated list of contextual factors From a synthesis of literature on the subject, the review evidence and related mechanisms): identified a number of key factors critical to realizing the full statements promise of ‘the collaborative advantage.’ 1. Large system transformation in healthcare systems requires both 1. Clear common aims. It often takes time and cycling top-down leadership that is passionately committed to change, as through the various stages of direction setting, action, and well as distributed leadership and engagement of personnel at all trust building, to develop the overarching partnership-level levels of the system. goal, common language, and aims necessary to enable and sustain a productive partnership. 2. Measurement and reporting on progress toward short- and long- 2. Trust. An essential foundation for any partnership, trust term goals is critical for achieving effective and sustainable large builds on itself over time, often starting with modest, low-risk system transformation. initiatives. 3. Consideration and acknowledgement of historical context will 3. Leadership. Effective interorganizational collaboration help avoid unnecessary pitfalls, and increase buy-in and support requires sustained and engaged leadership, and a shift in from system stakeholders. leadership/management style from ‘command and control’ to facilitating and empowering, and from delegation to participation. Saul et al. Implementation Science 2013, 8:103 Page 10 of 15 http://www.implementationscience.com/content/8/1/103 Table 2 Case studies (Continued) 4. Large system transformation in healthcare systems relies on 4. Sensitivity to power issues. Each partner in a significant physician engagement in the change process. collaborative brings different resources and strengths. Effective collaboration requires acknowledging these differences and taking care in negotiating expectations and the ground rules for decision-making. 5. Large system transformation that aims to increase patient- 5. Membership structures. Shared understandings about centeredness requires significant engagement of patients and what the collaboration involves, and formalized rules, roles families in the change process. and structures enable participation. Both governance and task structures are important. The evidence shows the need for effective coordination infrastructure with agreed-action strategies, and sufficient resources, capacity, and role clarity to support good communication and management functions. Because membership is often dynamic and changing, continuing work is essential to sustain the shared understanding and common focus. Effective coordination structures also help speed the uptake of innovations. 6. Action learning. Effective collaborations continuously improve through feedback loops and reflective shared learning. Strengths of Much of the success of this review was due to the timely topic, This review was positioned to inform a strategic planning the review: which inspired commitment from the reference group and expert initiative for OBSSR; as such it was an important support to panellists. Key to this was the direct involvement of key decision future work for them and was applied in practice almost makers in the Saskatchewan government, which ensured relevancy immediately. This review began with a key set of articles that of the final product and maximized the potential for uptake of the were provided by the research team and expert panellists, findings. The project was designed for and succeeded in blending thus expediting the process. Articles were added as we came practice knowledge with research knowledge. Within the design, across them through citation searching or from the panellists this review was unique in the use of a consensus network to over the duration of the project. As is the case in all validate the findings. This network was made up of national and successful RRRs, the expert panel was engaged and international experts in the area; using an online survey, they were committed to the topic. asked to comment on the five evidence statements and reflect on how well these statements compared with their understanding and experience. Out of 100 people invited, 50 % took part. Considering the voluntary nature and the length of the survey (it took people on average 30 minutes), we deemed this a very good response rate. The consensus network was particularly useful given the literature base was relatively sparse. Also contributing to the success of the project was the opportunity to link the final workshop with another major CIHR initiative, providing an additional opportunity for shared learning and partnering. Challenges As is often a reason for choosing a realist review methodology, the The review had to be done within a two-month timeframe for the amount of published and grey literature on the topic was scarce. with limited resources. This meant no time or money to review: Remedies to this challenge were in-depth and on-going include a key word literature search. However, due to the consultation with a reference group and an expert panel, further research team’s prior work and the expertise from the panel, enhanced by the use of the consensus network. An additional the project was initiated with a solid set of articles for review, challenge was the fact that the findings were intuitive, and did not and completed within the timeframe allocated for the appear to provide ‘novel’ solutions or courses of action to the project. decision makers in the Saskatchewan Ministry of Health. However, they were pleased to have formal confirmation of what they already knew. Considering tacit knowledge often runs ahead of research knowledge, this can be a typical of many reviews, rapid and otherwise. Both are designed to be responsive to local policy needs. process, allowing the entire review process to be com- Both are grounded in realist logic and constructs, expli- pleted within three to six months (see [11] for an ex- citly attempting to understand the interactions between ample). There are limitations associated with reducing the contexts, mechanisms and outcomes. However, they also time available to complete such a review, as are discussed have some key differences, which allow policy makers to below. However, some policy decisions are required in a apply each type of methodology strategically to maximize very short time frame (six months or less), thus making a its utility within a particular local constellation of history, methodology that can gather and synthesize the maximum goals, resources, politics and environment. In particular, amount of evidence in a compressed time period ex- the RRR methodology is explicitly designed to engage tremely useful. knowledge users and review stakeholders to narrowly de- The cost implications for the rapidity of the review fine the research questions, and to streamline the review process are difficult to quantify, since the bulk of our Saul et al. Implementation Science 2013, 8:103 Page 11 of 15 http://www.implementationscience.com/content/8/1/103 Table 3 Comparison of RRR and ‘traditional’ realist review methods RRR method Full realist review method ▪ Examine interactions between mechanisms, context and outcomes ▪ Examine interactions between mechanisms, context and outcomes ▪ Short-term (3- to 6-month turnaround) ▪ Longer term (12- to 24-month turnaround) ▪ Responsive to local policy needs ▪ Reponsive to local policy needs ▪ Results are utilization-focused ▪ Results are utilization-focused ▪ Includes both reference group and expert panel ▪ Likely includes a reference group (or something similar); there is no standard for involvement of an expert panel ▪ Involvement of expert panel allows for conclusions and ▪ Includes both published and grey literature, search process is recommendations to be developed from an emerging/nascent comprehensive (until the point of theory saturation), and program theories literature can be developed over a period of months. work is with RRR, thus prohibiting a good comparison (including those in the process of publication) will be within our own team for rapid vs. traditional realist re- identified and included. Secondly, review of preliminary views. We surmise that the skillset required for the re- findings by the expert panel validates results of the re- view and synthesis increases the cost for personnel, view, ensuring that the interpretation of the literature is although the specific relative costs will vary by project, consistent with the experiences of those currently en- content area, and individuals involved. While the overall gaged in work around the review topic. The intent of in- length of time for RRRs will decrease costs overall, pro- corporating an expert panel is to integrate the lessons ject management costs per month are likely to be greater learned from the literature with those of identified con- for RRRs than for longer realist reviews, given the neces- tent leaders who have experience working to integrate sity to interact frequently with the reference group and knowledge from research, practice and policy perspec- expert panel, and the need to obtain timely responses tives, and to build program theories based on that com- from them. bined knowledge. As discussed, the RRR method outlined here involves The RRR method also differs from a more traditional both a local reference group and an expert panel. A realist review in the role of theory, and specifically the traditional realist review likely engages a group equiva- presentation of theory in the final synthesis product. A lent to a local reference group, but to our knowledge no ‘traditional’ realist review explicitly grounds the review expert panel is engaged as a standard part of the realist process in theory; it produces a narrative summary of approach to knowledge synthesis. Among realist reviews the interactions between context, mechanisms, and out- more generally, ‘advisory groups’ are quite common, and come configurations [8]. The reviewer then constructs often serve the role of a merged ‘reference group’ and one or more ‘middle range theories’ (i.e., ones that ‘in- ‘expert panel’. We have found it useful to separate out volve abstraction… but [are] close enough to observed the ‘expert panel’ into its own functioning unit. To make data to be incorporated in proportions that permit em- the engagement process more efficient, asking the refer- pirical testing’ [19]) to account for the configurations. ence group to focus on the utility of the findings, and The literature is then explored to ‘test’ these theories their relevance to the work at hand within a local con- based on emerging findings from the review. In the RRR text, has worked well. The expert panel is similarly asked method, results are generated through a very similar to focus its efforts on the validity of the findings and our process; however, theory-building is viewed as a support- interpretation of the evidence. The engagement of ex- ive process to identifying families of interventions (I) perts allows for at least two important contributions: and explaining why they would produce outcomes of interest (O) by generating specific changes in context firstly, it provides a validation process for findings that have been generated from an expedited review process. (C) that would then trigger particular mechanisms, ra- In the RRR method, the search strategy is intended to be ther than the primary result. In some cases, however, the content area for the review is nascent enough that there robust, but not comprehensive. Due to the time con- straints, it is fully acknowledged that not all references is insufficient published literature to generate fully devel- related to the research question, or even all that will oped theoretical frameworks in which to ground the re- view. In those cases, the local reference and expert produce theoretical saturation, will be identified and in- cluded in the review. However, by engaging the expert panels assist in producing and refining program theories panel in the searching stage of the project, we can accel- that describe the current best thinking about how pro- grams should work in different contexts. In either case, erate the search process, and at the same time be rea- sonably assured that any key theoretical contributions the focus of the review team is to produce generally ac- within critical articles, reports, or other resources cepted explanations among policy makers and content Saul et al. Implementation Science 2013, 8:103 Page 12 of 15 http://www.implementationscience.com/content/8/1/103 experts for what works within a particular set of parame- available. For example, the evidence informed public ters, rather than aiming to produce explanations that are health policy and practice review consisted primarily of potentially transferrable across contexts and populations. a literature review and synthesis with no reference The distinction may appear to be semantic, but its impact group. On the other end of the spectrum of review com- is very real for both process and product of the review. plexity, the large system transformation review (KAST) included a much broader and more extensive consult- Strengths of the rapid realist review process ation with content experts. While this was not critical to Reviewing the similarities and differences between the the success or usefulness of the final report, it added to RRR and more traditional realist review methods high- the nuanced contextualization provided in the final re- lights several strengths of the RRR approach. Similar to port. Key to the KAST project, the report commissioners realist reviews more generally, the RRR method is time explicitly requested recommendations for government efficient, and may in fact be more amenable to time limi- action that could enhance the likelihood of success for tations than other types of realist reviews (due in part to large system transformation projects. Having such a re- the increased involvement of the expert panel). RRR is quest made it easier to elicit useful feedback from the grounded in local context, with explicit, extensive, iterative content experts that were surveyed for the project. engagement with a local reference group comprising rep- resentatives of potential ‘knowledge users’. The reference Limitations of the rapid realist review process group provides feedback on the project at critical times, While the RRR method has several strengths and advan- including when developing project scope and research tages, as noted above, there are also several limitations questions, identifying key literature, and interpreting re- to the RRR approach. sults (‘Do these findings and recommendations make The RRR method explicitly uses an expedited search sense for our context? Will we be able to use these re- process. This is an advantage with respect to the faster sults?’). As such, the RRR method is responsive to local turnaround time for the finished review, but it may also policy needs and is therefore adherent to the goals of a result in certain resources/references being missed, po- realist approach to knowledge synthesis [3]. As a result, tentially introducing a source of bias. However, the ef- while the RRR method is adaptable to scale, it tends to op- fects of this potential limitation are buffered by the erate most effectively when reviews are commissioned by engagement of the expert panel. Experts validate the those who have authority and power to enable change, findings to ensure that critical elements are not missed, where the policy questions to be answered map clearly and that nuances from emerging practice are included. onto the jurisdiction that is responsible for effecting policy The short time frame in which RRRs are typically change, and where key stakeholders from that jurisdiction conducted can make it difficult to fully theorize the who will be putting policy changes into practice are repre- mechanisms that are identified, as well as the interac- sented on the local reference group (as occurred in the tions between context, mechanisms and outcomes, as is KAST review; see Table 1). When reference groups in- at the core of a full realist review. This may conse- clude (or provide links to) key decision makers in commis- quently limit the generalizability and potency of findings. sioning agencies, this leads to closer connections between Therefore, policy makers interested in sharing results knowledge producers and users. more broadly may need to consider the inclusion of Similar to traditional realist reviews, the RRR method- other examples of when the same (or similar) mechan- ology also works well in situations where there is a small ism is in operation, though this might be an intervention but emerging body of evidence on which to base policy that has not been labelled as being the ‘same’ as the one decisions. The use of the expert panel allows for incorpor- under consideration. However, this is unlikely to be an ation of knowledge-in-the-making into policy recommen- impediment to the actual utilization of the review find- dations, as well as the practice knowledge of those who ings in practice. Working with the local reference group have been engaged in application of research findings to a throughout the project ensures that the project scope, policy context. For example, in the review of methods to research questions, and format of project outcomes will increase organizational capacity to address health literacy meet the needs of those who will use the information. issues within healthcare organizations, the actual pub- Given the time-intensive nature of the RRR review lished literature specifically relating to the review topic process, as well as the critical role of the expert panel, it was limited. As a result, the expert panel helped frame the is unlikely that government agencies or departments will findings significantly to be consistent with what they had have the in-house capacity to complete these reviews. experienced in practice. This enhanced the value of the Adding the RRR method to a policy maker’s toolkit will final report for the knowledge users. routinely require external input from expert teams. Finally, the RRR methodology allows for the ability However, this may actually be an advantage; it allows to scale the project based on the time and resources expert methodologists to be engaged at specific times in Saul et al. Implementation Science 2013, 8:103 Page 13 of 15 http://www.implementationscience.com/content/8/1/103 order to answer specific questions, thus targeting re- use of findings particularly relevant. These threats are sources strategically. One critical advantage is the ex- three fold: periential learning that results, deepening the systems thinking skills required for effective change. The relative 1. Lack of alignment between the policy question and strengths are brought to bear in ways that maximize the the composition/jurisdiction of the reference group. allocation of resources, time and talents, making such a If those engaged as the local reference group aren’t process efficient as well as effective. the ones who will be able to use the results, the Forming the membership of the reference group and process will have significant challenges. ensuring sustained involvement are challenges for all re- 2. Lack of sufficient time to conduct the review, or views, particularly in rapidly changing political environ- barriers that prevent the process from proceeding ments where membership of the reference group may according to schedule. As discussed above, rapid change during the course of a given project. However, feedback from both reference group and expert the role of the reference group in the RRR methodology panel members is necessary for conducting rapid allows key agency or government staff to be engaged in realist reviews. If competing time commitments the process without requiring excessive time commit- prohibit such a level of engagement, the project will ments. Similarly, the engagement of expert panels does lose its ability to produce results in a timely fashion, rely on panellist availability and willingness to contribute the quality of the review may be reduced, or the their time and ideas. Providing honoraria for expert utility of the results may be lessened. panellists, and/or providing opportunities for them to be 3. Lack of continued engagement and evaluation of the involved in dissemination of the work through confer- impact of RRRs. Ideally, RRRs would generate ence presentations, papers, or other avenues, can make evidence-based recommendations for policy action. it easier to engage the right experts at the right time. These actions would be implemented and evaluated to A final potential weakness can result from the require- determine their impact. Additional work needs to be ment that all members of the review team be well versed done to demonstrate and document the impact and in realist review methodology, while at the same time value of the RRR methodology. A lack of engagement commissioners of realist reviews (such as policy makers) or interest in these areas by policy makers will make it do not share the same deep understanding of the meth- more difficult to continue to make the case for the use odology and its value for policy. In our experience, pol- of such methods in the future. icy makers tend to be more concerned about what they are able to accomplish within one specific context (e.g.,a Future work province) rather than adding to the knowledge-base of Future work is needed in several areas to support and middle-range theories. While they need to understand enhance the value of the RRR methodology in deve- C-M-O interactions to be able to identify potentially ef- loping evidence-informed policy. First, as noted above, fective interventions, their focus tends to be more in additional measurement is needed to document the out- intervention (I) – outcome (O) links. We see a growing comes and impact of RRRs. Conducting qualitative re- interest in the how and why interventions work or don’t search and evaluation with the users of products work for various audiences, but understanding the pro- developed using RRR methods will add enormously not gram theories that underlie success or failure of inter- only to the field of implementation science, but will also ventions tends to be lower on the list of priorities than develop a more robust toolkit for policy makers facing producing results. As a result, our reports have had to decisions in a context of uncertainty. Second, and relat- highlight I-O links, and use the program theories we de- edly, work is needed to develop robust criteria with veloped to help explain why those interventions might which the impact of RRR methods can be assessed. work well in a given context for specific populations. Third, a method is needed to explicitly document the boundaries for findings produced with RRR methods. While results of RRRs typically are presented to stake- Potential risks to the impact of rapid realist reviews holders within one specific context, a method is needed The RRR methodology was explicitly developed to bet- for identifying the limits of the review in terms of time ter meet the time-sensitive needs of policy makers for period, geographical area, populations, and environmen- coherent, focused and relevant syntheses of knowledge tal factors that might limit the generalizability of results. to develop evidence-informed policies. There are, how- Finally, a structured program of evaluation needs to be ever, threats to the ability of intended knowledge users developed that guides this process and that is helpful to to put the findings into practice. While these risks are users as well as review methodologists. Currently, all as- applicable to any similar review, the underlying pur- sessments of impact are largely based on user feedback. pose of the RRR methodology makes threats to the Following completing of reviews, a lag-time is required Saul et al. Implementation Science 2013, 8:103 Page 14 of 15 http://www.implementationscience.com/content/8/1/103 to enable actual action on review recommendations. The In sum, the authors may receive financial gain in the future from the publication of this manuscript. In our view, the conflict between academic ideal timeframe, process, and methods for assessing im- and business interest in this area is unavoidable, and bridging the gap pact and outcomes are areas for future research and between these interests is vital to supporting the research to policy and collaboration. practice process. The evidence is clear that knowledge uptake is poor without effective structures to support the process. Conclusions Authors’ contributions Rapid Realist Review offers a useful tool for addressing JS made substantial contributions to: the acquisition, analysis and synthesis of data presented in Table 1; designing, drafting and revising the manuscript. time-sensitive policy decisions. The RRR methodology CW made substantial contributions to: the design, drafting and revision of provides a way of handling the tight time limits and the manuscript; analysis and interpretation of data in Table 1. JB made rapid turnaround necessary that often characterize pol- substantial contributions to: the acquisition, analysis and synthesis of data presented in Table 1; drafting and revising the manuscript. AB made icy decisions. The risks to the rapid nature of the review substantial contributions to: the analysis and synthesis of data presented in methodology are a potential lack of comprehensiveness, Table 1; drafting and revising the manuscript. All authors have approved the or an increased risk of bias. These risks are compensated final version of this manuscript. for by the involvement of content experts who can en- sure that the results are complete, and that they resonate Acknowledgements with practice knowledge. RRR also incorporates local The authors wish to acknowledge Professor Patricia O’Campo and Dr. Geoff Wong for thoughtful (and in the case of Dr. Wong, repeated) reviews of this knowledge users and links to commissioning agencies, manuscript. The authors wish to thank all agencies listed in Table 1 who thus enhancing both usability and utilization of results. have collaborated on previous rapid realist reviews. The authors also wish to Rapid Realist Reviews are scalable with respect to in- thank all associated team members who have contributed to rapid realist reviews outlined in Table 1. volvement of external experts based on the time and re- sources available. The RRR methodology works well in Author details contexts where there is limited evidence due to its in- InSource Research Group, 982 Thermal Dr., Coquitlam, BC V3J 6S1, Canada. North American Research & Analysis, Inc, 1016 11th Avenue NE, Faribault, corporation of practice evidence via expert panels. MN 55021, USA. Centre for Clinical Epidemiology and Evaluation, Vancouver While the end products of RRRs focus less on theory Coastal Health Research Institute, The University of British Columbia, development than on intervention-outcome links, they Research Pavilion, 7th Floor, 828 West 10th Avenue, Vancouver, BC V5Z 1M9, Canada. School of Population Health, The University of Adelaide, Level 7, still rely on program theories to explain how and why 178 North Terrace, Adelaide, SA 5005, Australia. InSource Research Group, certain families of interventions may produce outcomes 6 4581 Anhalt Rd, Kelowna, BC V1W 1P7, Canada. InSource Research Group, of interest within a given context for specific popula- 6975 Marine Drive, West Vancouver, BC V7W 2T4, Canada. School of Population and Public Health, University of British Columbia, 2206 East Mall, tions. It can be more difficult to engage policy makers in Vancouver, BC V6T 1Z3, Canada. theory development, but this does not lessen the utility of realist reviews for producing evidence-informed pol- Received: 18 January 2013 Accepted: 24 August 2013 Published: 5 September 2013 icy. Government agencies interested in using a RRR methodology will likely need to involve expert method- ologists. However, this helps maximize the efficiency of References 1. Lavis J, et al: Towards systematic reviews that inform health care key agency or government staff. management and policy-making. J Health Serv Res Policy 2005, In conclusion, for policy makers faced with making 10(Suppl 1):35–48. difficult decisions in short time frames, related to 2. Pawson R: Evidence-based policy: in search of a method. Evaluation 2002, 8(2):157–181. system-level programs/initiatives/directions, for which 3. Pawson R: Evidence-based policy: a realist perspective. London: Thousand there is sufficient (if limited) published/research and Oaks, Calif.: SAGE; 2006:196. practice-based evidence available, RRR provides a prac- 4. Pawson R: Evidence-based policy: the promise of realist synthesis. Evaluation 2002, 8:340–358. tical, outcomes focused knowledge synthesis method. 5. Green LW: From research to “best practices” in other settings and populations. Am J Health Behav 2001, 25(3):165–178. Competing interests 6. Greenhalgh T, et al: Diffusion of innovations in service organizations: The Rapid Realist Review methodology described in this paper is designed systematic review and recommendations. Milbank Q 2004, 82(4):581–629. to bridge multisectoral barriers to knowledge use. Although the authors 7. Astbury B, Leeuw F: Unpacking black boxes: mechanisms and theory have academic interest in this methodology, they also engage in the building in evaluation. American Journal of Evaluation 2010, 31:363–381. methodology as consultants on a range of projects. Some of these projects 8. Pawson R, et al: Realist review–a new method of systematic review are funded by grants managed by universities, some through contracts to designed for complex policy interventions. J Health Serv Res Policy 2005, the InSource Research Group. InSource was created to provide a vehicle for 10(Suppl 1):21–34. responding to policy maker needs for knowledge synthesis in a more timely 9. Ganann R, Ciliska D, Thomas H: Expediting systematic reviews: methods way than is possible through the normal grant funding process. and implications of rapid reviews. Implement Sci 2010, 5:56. In the interests of transparency and to address conflict of interest concerns, 10. Watt A, et al: Rapid versus full systematic reviews: validity in clinical it is very possible that publication of this paper could enhance InSource’s practice? ANZ J Surg 2008, 78(11):1037–1040. reputation and could result in future contracts for the company. It must also 11. Riley B, Norman C, Best A: Knowledge integration in public health: a rapid be noted that two of the four authors (JB and AB) are InSource directors, review using systems thinking. Evidence and Policy 2012, 8:417–431. and the other two (JS and CW) have worked, or in the future may work, 12. Khangura S, et al: Evidence summaries: the evolution of a rapid review under contract to InSource. approach. Syst Rev 2012, 1(1):10. Saul et al. Implementation Science 2013, 8:103 Page 15 of 15 http://www.implementationscience.com/content/8/1/103 13. Arksey H, O’Malley L: Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, Theory and Practice 2005, 8(1):19–32. 14. Wong G, et al: RAMESES publication standards: realist syntheses. BMC Med 2013, 11:21. 15. Lavis JN, et al: Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med 2012, 9(3):e1001186. 16. Keown K, Van Eerd D, Irvin E: Stakeholder engagement opportunities in systematic reviews: knowledge transfer for policy and practice. J Contin Educ Health Prof 2008, 28(2):67–72. 17. Van de Ven AH, Johnson PE: Knowledge for theory and practice. Acad Manage Rev 2006, 31:802–821. 18. Van de Ven AH: Engaged scholarship: a guide for organizational and social research. Oxford: Oxford University Press; 2007. 19. Merton RK: On theoretical sociology; five essays, old and new. A free press paperback. New York: Free Press; 1967:180. doi:10.1186/1748-5908-8-103 Cite this article as: Saul et al.: A time-responsive tool for informing policy making: rapid realist review. Implementation Science 2013 8:103. Submit your next manuscript to BioMed Central and take full advantage of: • Convenient online submission • Thorough peer review • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Implementation Science Springer Journals

A time-responsive tool for informing policy making: rapid realist review

Download PDF
15 pages

Loading next page...
 
/lp/springer-journals/a-time-responsive-tool-for-informing-policy-making-rapid-realist-baZ9bLShVy

References (43)

Publisher
Springer Journals
Copyright
Copyright © 2013 by Saul et al.; licensee BioMed Central Ltd.
Subject
Medicine & Public Health; Health Promotion and Disease Prevention; Health Administration; Health Informatics; Public Health
eISSN
1748-5908
DOI
10.1186/1748-5908-8-103
pmid
24007206
Publisher site
See Article on Publisher Site

Abstract

Background: A realist synthesis attempts to provide policy makers with a transferable theory that suggests a certain program is more or less likely to work in certain respects, for particular subjects, in specific kinds of situations. Yet realist reviews can require considerable and sustained investment over time, which does not always suit the time-sensitive demands of many policy decisions. ‘Rapid Realist Review’ methodology (RRR) has been developed as a tool for applying a realist approach to a knowledge synthesis process in order to produce a product that is useful to policy makers in responding to time-sensitive and/or emerging issues, while preserving the core elements of realist methodology. Methods: Using examples from completed RRRs, we describe key features of the RRR methodology, the resources required, and the strengths and limitations of the process. All aspects of an RRR are guided by both a local reference group, and a group of content experts. Involvement of knowledge users and external experts ensures both the usability of the review products, as well as their links to current practice. Results: RRRs have proven useful in providing evidence for and making explicit what is known on a given topic, as well as articulating where knowledge gaps may exist. From the RRRs completed to date, findings broadly adhere to four (often overlapping) classifications: guiding rules for policy-making; knowledge quantification (i.e., the amount of literature available that identifies context, mechanisms, and outcomes for a given topic); understanding tensions/ paradoxes in the evidence base; and, reinforcing or refuting beliefs and decisions taken. Conclusions: ‘Traditional’ realist reviews and RRRs have some key differences, which allow policy makers to apply each type of methodology strategically to maximize its utility within a particular local constellation of history, goals, resources, politics and environment. In particular, the RRR methodology is explicitly designed to engage knowledge users and review stakeholders to define the research questions, and to streamline the review process. In addition, results are presented with a focus on context-specific explanations for what works within a particular set of parameters rather than producing explanations that are potentially transferrable across contexts and populations. For policy makers faced with making difficult decisions in short time frames for which there is sufficient (if limited) published/research and practice-based evidence available, RRR provides a practical, outcomes-focused knowledge synthesis method. Keywords: Realist review, Rapid review, Evidence synthesis, Health policy * Correspondence: [email protected] Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute, The University of British Columbia, Research Pavilion, 7th Floor, 828 West 10th Avenue, Vancouver, BC V5Z 1M9, Canada InSource Research Group, 6975 Marine Drive, West Vancouver, BC V7W 2T4, Canada Full list of author information is available at the end of the article © 2013 Saul et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Saul et al. Implementation Science 2013, 8:103 Page 2 of 15 http://www.implementationscience.com/content/8/1/103 Background a high level of training and experience is required, which Efforts to improve the connections between research may not be found routinely in government or policy devel- evidence and policy decisions have resulted in a number opment agencies, academic institutions, or community- of sophisticated synthesis tools [1]. Pawson has broadly based organizations. Perhaps most relevant to our current classified evidence synthesis approaches as being meta- discussion, realist reviews as outlined by Pawson et al. analytical (re-analysing data from numerous studies to [3,8] can require considerable and sustained investment arrive at broadly generalizable findings) or narrative over time, which does not always suit the time-sensitive (using text-based data extractions to compare study demands of many policy decisions. In addition, due to findings as a way for understanding why programs or in- their expansive and exploratory nature, realist reviews can terventions have certain effects), both of which are com- often suffer from ‘scope creep’. monly employed for informing policy decisions [2]. ‘Rapid reviews’ have emerged in response to the incom- While Pawson notes the relative advantages of meta- patibility between information needs of policy makers and analytical and narrative approaches, each technique also the time requirements to complete systematic reviews. has key limitations: meta-analyses fail to provide under- Rapid reviews provide a way to generate similar types of standing of how programs work, while narrative reviews knowledge synthesis as more comprehensive systematic give little insight into how a program may operate in reviews do, but in a much shorter time period. While some different settings or under different circumstances [3,4]. have questioned the validity of rapid reviews [9,10], there Limited generalizability of study results for a given remains a need to achieve a balance between comprehen- context has been addressed as a common concern in siveness and timeliness for many policy-relevant decisions evidence-informed public health policy [5,6]. Alterna- [11]. Therefore, understanding how this realist approach tively, we have realist reviews, which are an approach to may be applied in cases where there is limited time and re- synthesising evidence first promoted by Pawson [4]. sources to generate a realist-based product that can in- Pawson notes that the underlying principles of realist corporate research, theory, and practice knowledge and approaches are the links between interventions (I), con- thus meet the demands of real-time policy developers/ texts (C), mechanisms (M), and outcomes (O) [4]. In evaluators, is a valuable contribution to both evidence- contrast to specific activities included as part of an inter- informed policy-making and the realist review literature. vention or program, mechanisms (M) are thought to be ‘Rapid Realist Review’ methodology (RRR) has been de- the responses by people that are triggered by changes in veloped by members of our group as a tool for applying a context (C) [7]. Traditional realist syntheses use these realist approach to a knowledge synthesis process and C-M combinations to generate transferable ‘program producing a product that is useful to policy makers in theories’ that suggest that certain interventions are more responding to time-sensitive and/or emerging issues or less likely to work in certain ways, for certain people, where there is limited time and resources. Where relevant, in certain situations [3,4]. In other words, how does a we compare the RRR methodology to realist reviews gen- change in context generate a particular mechanism that erally, or ‘traditional’ realist reviews. While there is no set in turn produces specific outcomes? definition for what constitutes a realist review other than In developing these program theories, realist reviews its use of realist logic and constructs, we present ‘trad- assume a broad inclusion of evidence, both quantitative itional’ realist reviews as those that typically engage in a and qualitative, which is used to develop an understand- much longer exploration of the literature and ‘testing’ of ing of ‘what works, for whom, in what contexts, to what theories, as well as those that present results within a extent, and most importantly how and why [2,3]?’ framework of theory development. While others have Therefore, instead of offering prescriptive advice on begun to describe methodologies for short-term evidence what the ‘best buys’ might be for policy makers, a realist synthesis projects [12], the RRR methodology described synthesis attempts to provide a transferable theory that here is intended to incorporate the theory specification of suggests that a certain program is more or less likely to a realist review [8] and the boundary clarification aim of a work in these respects, for these subjects, in these kinds scoping review [13]. (See [11] for an example of an earlier of situations [2,3]. review). This is also consistent with the recently published However, realist review methods are not without their RAMESES (Realist And MEta-narrative Evidence Synthe- difficulties. The iterative, flexible nature of realist reviews ses: Evolving Standards) publication standards for realist does not align well with protocol-driven, standardized pro- syntheses, which note that realist reviews need to be fo- cesses common to established systematic review methods cused based on the time and resources provided as well as [8]. As results from a realist review are context dependent, the questions that need to be answered [14]. the generalizability of findings to other settings will depend In our experience, policy makers have been more in- on the operation of similar mechanisms to generate out- terested in knowledge syntheses that highlight possible comes of interest. Moreover, in completing a realist review, interventions (I) that could be implemented within a Saul et al. Implementation Science 2013, 8:103 Page 3 of 15 http://www.implementationscience.com/content/8/1/103 specific context (C) that in turn interact with various incorporated it within the RRR method explicitly as a mechanisms (M) and produce outcomes (O) of interest. means to streamline the review process. Other variations As a result, the emphasis of our reviews has changed over that are distinct from more traditional realist reviews are time from one focused on producing transferable theories, noted below. to one focused on identifying groups of interventions re- lated to outcomes of interest for policy makers. Our modi- 1. Development of the project scope: clarifying with fied review methodology ‘works backwards’ from the the knowledge users the content area of interest for desired outcome (e.g., large scale transformation of health the review. As with any type of realist review, this systems, sustainable cultural change, etc.), to ‘families of step is critical in ensuring a feasible review process, interventions’ (I) that can be implemented to produce regardless of the desired timeline. those outcomes, supported by a theoretical understanding 2. Development of specific research questions: once of the contexts (C) within, and mechanisms (M) by which the project scope has been narrowed, discussing the such interventions operate. In doing so, the RRR method- specific questions that knowledge users are most ology focuses less on the development of theory that is interested in answering. Refining these questions to transferable across domains, than on the distillation of ensure that there is enough evidence to be able to theory-driven, contextually relevant interventions that are answer them, at least in part. As with step 1 above, likely to be associated with specific outcomes within a par- this is a critical component of any type of realist ticular set of parameters. review. Here, we describe our experience in developing RRR, 3. Identification of how the findings and the key ingredients in making it work, the policy ques- recommendations will be used: this includes tions to which it is most suited, the strengths and limita- formulating a purpose statement that helps identify tions of the method, and the future directions of how how the findings of the review will be used by the this approach may be optimized in practice. In doing so, target audience. Utilization of review products is a key we provide key examples of RRR projects completed by element in the RRR methodology, and one which our group over the past six years. systematic reviews often do not address. While traditional realist reviews may incorporate this step as Methods well, it is not explicitly included in the RAMESES We provide here brief summaries of key examples of re- publication standards for realist reviews [14]. This views completed to date by this team. Table 1 includes could be considered part of step 2 above, although it the commissioning agencies, title, duration and purpose has allowed us to focus our reviews more clearly with of each review. For this paper, we also considered key the knowledge users in mind throughout the findings and recommendations resulting from each, as searching, extraction, and synthesis process. well as the strengths and challenges that arose while 4. Development of search terms: collaboratively completing the review. We draw from these examples to identify terms likely to be relevant to the project highlight the conceptual strengths and limitations of scope, purpose, and research questions. 5. Identification of articles and documents for inclusion rapid realist review methods and the circumstances under which they offer most value. in the review (both published and grey): begin with a list of documents as identified by knowledge users and content experts. In addition, use the search Methods of realist review As Pawson has described elsewhere [3], key stages and terms to iteratively generate lists of documents that may be included in the review. tasks for a realist review include the following steps: iden- 6. Quality review: narrow the search terms based on tifying the review question, searching for primary studies, quality appraisal, extracting the data, synthesizing the data, the results that are most relevant to the review topic. Simultaneously, poll the knowledge users and and disseminating the findings (see [3], chapter 4). How- external content experts to identify documents ever, while Pawson has stated that realist reviews need to be focused, the methodology he has described has not (published and grey) of key importance for the review. We explicitly acknowledge that a search been explicit about how to focus the review, particularly using the RRR methodology will not be for cases where a policy recommendation is required comprehensive. Polling knowledge users and content quickly. In response to the need for a time- and resource- experts to identify key articles accelerates our search sensitive review process, we have modified and expanded process. This, combined with the validation step on Pawson’s original methodology, including the explicit (#8 below), helps ensure that we are not missing engagement of content experts drawn from stakeholders. significant sections of the literature with our While engagement of experts and stakeholders is a com- abbreviated search. mon strategy for traditional realist reviews, we have Saul et al. Implementation Science 2013, 8:103 Page 4 of 15 http://www.implementationscience.com/content/8/1/103 Table 1 Examples of rapid realist reviews undertaken by our group Commissioning agency Title Purpose statement Duration Public Health Agency of Canada Interventions To Increase Organisational The purpose of this project was to conduct a Six months Capacity To Address Health Literacy (2012) realist review of available literature (published and grey) to summarize what we know and what we don’t know about the development of capacity within organizations to plan, implement, and sustain health literacy interventions. This review did not assess the relative efficacy of health literacy interventions themselves, but rather, attempted to identify what mechanisms and contextual factors improve the capacity of healthcare organizations to address health literacy in the individuals they serve by planning, implementing, evaluating and sustaining interventions. Canadian Institutes of Health Knowledge And Action For System The Saskatchewan Ministry of Health has a Six months Research (Partnering organization - Transformation (KAST): A Systematic Realist mandate to support a significant Saskatchewan Ministry of Health) Review and Evidence Synthesis of the Role of transformation of the provincial health system. Government Policy in Directing Large System Large system transformation refers to Transformation (2010) systematic initiatives to create coordinated change across organizations working toward shared priorities within specified boundaries. The Ministry recognizes that success hinges on a cultural change based on collaboration, a comprehensive innovation strategy for system redesign, and systems integration. Therefore, they seek guidance -- on such considerations as successful models and strategies, partnership principles (including with patients), monitoring and evaluation -- from a systematic review of knowledge on large system transformation. With all of these considerations, there is an emphasis on the role of government. HealthLink BC and QUILTS Tele-health Contributions to Emergency A rapid review of tele-health contributions to Five months Department and Discharge Operations (2009) Emergency Department (ED) and discharge operations was conducted to: 1. Describe the patient population in selected local ED facilities; identify major drivers of ED utilization, throughput and follow-up; and conduct a rapid review of the effectiveness of tele-health strategies for improving ED/ discharge system performance. 2. Scan best practices in other jurisdictions and identify potential opportunities for HealthLink BC to add value for regional health authority ED/discharge systems. 3. Produce a model of system operations and a framework for ways in which HealthLink BC tele-health services might help to improve performance for ED services, and for discharge planning and follow-up. Public Health Agency of Canada Evidence-informed public health policy and This RRR was commissioned to summarize Two months practice through a complexity lens (2007) what is known about systems to support evidence-informed policy and what is not known. The review focused on frameworks most applicable to public health, with an emphasis on chronic disease prevention and healthy living. The review process also initiated an international collaboration with a focus on system development for evidence-informed public health. Saul et al. Implementation Science 2013, 8:103 Page 5 of 15 http://www.implementationscience.com/content/8/1/103 Table 1 Examples of rapid realist reviews undertaken by our group (Continued) HealthLine Services BC Tele-health in Support of Chronic Disease The purpose of this project was to provide Six months Management (2007) HealthLine Services BC (HLSBC) with a review of evidence and best practices for health line support for chronic disease management. Vancouver Coastal Health Population-based Framework for Chronic The overall goal of the project was to identify Three months Disease Self-management Support (2006) the most current view on effective models for self-management and self-management support, and to provide decision makers in the policy and practice community at Vancouver Coastal Health with a practical understanding of evidence-based interventions, within a population-based framework for self- management support. NIH Office of Behavioral and Social Interorganizational partnerships (2006) To provide a RRR of evidence-based principles Two months Science Research and the for effective interorganizational collaboration Canadian Health Services Research and partnership, as the principles relate to two Foundation contexts: the collaborations in which health research funders engage; and the forms and processes of linkage within universities and their community partners to promote interdisciplinary health research and effective knowledge exchange. The review also summarized the existing evidence to identify aspects of interorganizational partnerships (a) where sufficient evidence exists to conduct a full systematic review, and (b) where insufficient evidence exists to articulate the further primary research that is necessary. 7. Extraction of data from the literature: using an recommendations, further knowledge gathering and extraction template, pull out elements from synthesis, or evaluation of knowledge application. documents that can help answer the research Because findings are typically presented to policy questions. Data are extracted using identical makers with a focus on families of interventions that methods to a traditional realist review. Findings are can be implemented within their context, they can analyzed to build a form of realist program theory be easily acted on. Program theories are presented as that addresses the agreed focus and scope of the a way to understand how changes in context may review. As stated earlier, policy makers tend to be interact with mechanisms to produce outcomes of more interested in families of interventions that can interest, rather than as the primary findings. change context (C) in ways that interact with Program theories also help describe potential mechanisms (M) to produce outcomes (O) of unintended consequences resulting from changes in interest, and less interested in program theories. context and their resultant interactions with However, development of these program theories is mechanisms. essential for understanding the C-M interactions that can then help explain why these families of These ten steps are similar to the procedure for a interventions will or won’t work in the policy Cochrane or other systematic review, although they fun- maker’s setting. damentally differ due to their iterative nature (i.e., steps 8. Validation of findings with content experts: once 1 through 7 may be revisited iteratively throughout the program theories have been generated, they are RRR process) and the rapidity with which they are reviewed by content experts who have direct conducted. The primary differences involve the realist experience in the field to ensure that they represent philosophical approach to how the extraction, analysis the learnings of practitioners, and to fill any gaps and synthesis are completed (focusing on how context that may have been left by the published literature. interacts with mechanisms to produce outcomes), and 9. Synthesis of the findings in a final report. The report the involvement of knowledge users in shaping the final is formatted in a way intended to meet the needs of product. the knowledge users, based on the results of step 3 It is increasingly a necessity to engage knowledge users above, and the findings produced by steps 7 and 8. in order to receive funding to conduct reviews of all 10.Dissemination of results: working with the types. This fits well with the RRR design, and also con- knowledge users to apply the findings through policy stitutes one of its critical strengths. In particular, the Saul et al. Implementation Science 2013, 8:103 Page 6 of 15 http://www.implementationscience.com/content/8/1/103 involvement of a local reference group and expert panel scratch, identification and engagement of a group of con- facilitate both more efficient identification of key mate- tent experts can be enormously time consuming. How- rials to include in the review, and sufficiently robust ever, we have been able to circumvent the process of findings. Involvement of knowledge users and external identification, introduction, invitation and engagement experts also increases both the usability of the review due to our ability to draw on first- and second-degree net- products, as well as their link to current practice. Our work connections. engagement of a local reference group and an expert The initial search places a special emphasis on grey lit- panel is consistent with the recommendations put forth erature that may not be easily accessible through trad- by Lavis et al. to create a ‘dialogue that allows the data itional search methods, especially for cases in which the and research evidence … to inform and be considered published literature on the selected topic may be of lim- alongside the views, experiences, and tacit knowledge of ited relevance to the specific research question at hand. those who will be involved in, or affected by, future deci- Inclusion and exclusion criteria are developed to assist sions about the health system problem’ [15]. with selection of documents to review further. Based on At all stages of the process, a local reference group is these criteria, a screening tool is developed that is used engaged to ensure that the project will produce results to screen abstracts that are identified using the initial that will be relevant for the context in which they will search terms. As the search progresses, search terms and be used. The reference group typically includes repre- inclusion/exclusion criteria are modified iteratively with sentatives of the funding organization, as well as know- the goal of identifying the documents most relevant to ledge users (the target audience for the findings of the the review topic without missing many that might be review). As Keown et al. have shown, the potential bene- important to include. The expert panel is asked to con- fits of including stakeholders in the process of a review tribute a list of the most relevant articles at the start of include increased relevance, clarity and awareness of re- the search to streamline the process. The importance of view findings [16]. Van de Ven [17,18] provides a similar the expert panel’s and reference group’s help in the view for coproduction that he calls ‘engaged scholarship’. search process cannot be underestimated with respect to Given the explicit intention of the RRR methodology to conducting a rapid review. provide results that can be used by policy makers, these Once an initial list of documents has been identified, benefits become critical to the success of a project. pilot extractions are conducted with all members of the In addition to the local reference group, the RRR meth- review and extraction team. The extraction template in- odology identifies an expert panel made up of researchers cludes fields such as ‘theoretical foundation or concep- and practitioners, actively engaged in conducting work in tual framework’, ‘facilitators/barriers to success of the the content area for the review, who are in the process of project’ (e.g., contextual changes that helped or hindered negotiating the interplay between research, practice and the triggering of mechanisms to produce both intended policy. Engagement of an expert panel provides several and unintended outcomes), and ‘interactions between benefits, including focusing the review scope; streamlining context and mechanisms’. Template fields are modified the search process; filling gaps in the initial problem defin- and tailored to meet the needs of each individual review ition, search process or synthesis of findings; and ensuring project. A series of calibration exercises consist of com- appropriate interpretation of results. paring the extractions of the various team members for While these experts are often scattered internationally, level of detail, identification of relevant themes, and in- several factors have allowed us to rapidly convene such dividual style for presentation of findings. Between three panels within a matter of weeks from project initiation. and five articles are typically reviewed in this fashion. First, knowledge users often have connections to one or Further extractions are conducted by individual team more content experts for the topic, given their interest members. Additional consultations are conducted as and involvement with the topic. During initial discussions needed throughout the review process. As documents of project scope and research questions, collectively we are extracted, forward and backward citation searches identify experts who already have connections to the pro- are conducted on key articles. Additional documents are ject team or knowledge users. Initial invitations are ex- added to the review list throughout the process. Because tended to this small group who are fit for purpose for a the timeline for each rapid review is typically tight (six given review. Additional targeted invitations are extended months or less), there is a limited window of opportun- to those publishing within the domain of interest. Interest ity to add documents to the review. This emphasizes the and engagement can often be enhanced by further invi- importance of accelerating the review process with sev- tations coming from, or mentioning, experts who have eral key documents that can be used for forward and already agreed to participate. Finally, the time commit- backward searching. ment required by expert panellists is deliberately mini- After the initial round of extractions is conducted, the mized, and highlighted in the invitation. If starting from review team summarizes key themes and findings, Saul et al. Implementation Science 2013, 8:103 Page 7 of 15 http://www.implementationscience.com/content/8/1/103 focusing on contextual factors and mechanisms that im- positivist, linear, cause-and-effect perspective, but rather pact outcomes, and how context and mechanisms interact. from the perspective that mechanisms for change will al- Typically, a matrix is created listing contextual factors and ways be context-dependent. It is this grounding in the mechanisms. Findings are grouped by similar contextual realist philosophical tradition that distinguishes realist changes and how they trigger mechanisms to produce out- reviews from other types of systematic reviews. At the comes (demi-regularities). These groupings are regularly same time, it is essential that the review team have a shared and discussed within the review team to ensure clear sense of the scope and purpose of the review to en- validity and consistency in the inferences made, although sure maximum utility and application of the results for due to the time restrictions, this typically occurs twice or each project’s unique audience. Familiarity with the con- three times during the review. Findings are reviewed sim- text in which the results will be used (e.g., policy devel- ultaneously with the local reference group. Based on the opment) is also helpful. preliminary findings, the project questions, inclusion and Team composition differs for an RRR as compared to exclusion criteria, and the fields for the extraction tem- a traditional realist review in that the project manager plate are adjusted as needed. Following this process, the must be disciplined and able to keep a large complex extraction process is completed, and a draft of key findings process moving along according to schedule. The local is presented to both the local reference group and expert reference group and expert panel must be prepared to panel. Feedback is obtained to ensure relevance to the provide feedback in a timely fashion, as this can often be local context of knowledge users, consistency with the a limiting factor in how quickly a review can be com- body of published and grey literature, and consistency pleted. And finally, the synthesis lead must be able to re- with the current practice and tacit knowledge of the con- view large quantities of information efficiently, distil it tent experts. Feedback is used to expand on the findings into easily digestible components for review by the refer- and tailor the synthesis for the final report. ence group and expert panel, and quickly incorporate Team members needed to complete a Rapid Realist feedback while keeping the intended uses of the final Review include: product in mind. The reference group (knowledge users) must also be clear about their priorities, which can 1. Project manager, responsible for preparing internal help the review team make decisions about the review project documents, coordinating the dialogue and process, particularly as the project deadline approaches. managing a pre-determined set of requests for the Panel involvement varies in intensity over time. Clarity reference group and expert panel (providing of purpose and timelines, and a willingness of panellists feedback at each stage of the process from question to respond within the allocated time frame, are essential development to review of the final report), components of a project as well. consolidating feedback, maintaining the timeline, budget and other duties; Results 2. Local Reference group (including client Table 1 provides details of the RRRs completed by this representatives) and expert panel (ideally four to six team to date. Agencies commissioning RRRs have in- individuals for each group); cluded Federal Government Agencies (e.g., Public Health 3. Librarian (or information specialist) to lead on Agency of Canada, U.S. National Institutes of Health), document searches; Provincial Health Ministries (e.g. Saskatchewan Ministry 4. Review team (two to four individuals who screen of Health) and associated initiatives (e.g., Healthlink BC), abstracts, read selected documents, and perform regional health authorities (e.g., Vancouver Coastal Health extractions); Authority), and key research funding bodies (e.g., Canadian 5. Synthesis lead to oversee the review process and Institutes of Health Research). In a number of instances, play a main role in synthesizing information; reviews have been funded by research granting agencies 6. Academic or research lead. for work to be completed in partnership with identified collaborators. The composition of the review team is critical for the There is considerable variation in the topical focus of RRR methodology. All team members must be well- each RRR included in Table 1. Reviews often have been versed in the realist philosophy that serves as the theor- commissioned to synthesize information on complex, sys- etical underpinning for realist reviews. In addition, the tem oriented issues, such as large system transformation, academic or research lead and synthesis lead must not improving organizational competencies for addressing only be familiar with realism and realist approaches to health literacy needs, enhancing interorganizational part- synthesis reviews, but must also have participated in a nerships, and improving evidence-informed health policy realist review process previously. It is critical that the re- as well as bettering the people-centredness of health au- view team members approach the project not from a thorities. In addition, a number of reviews have focused Saul et al. Implementation Science 2013, 8:103 Page 8 of 15 http://www.implementationscience.com/content/8/1/103 on specific clinical specialties, domains or services, such confirming the superiority of multi-faceted chronic disease as tele-health interventions, emergency department dis- prevention programs compared to non-multi-faceted ap- charge practices, and optimal care models for chronic dis- proaches within a specific context for particular popula- ease management programs. tions, or identifying that large scale changes in Canadian As outlined in Table 1, RRRs may be completed for a health systems at this moment in time require engaged variety of purposes. Two broad purpose domains appear to physician and patient groups. Because they highlight I-C-M emerge, with RRRs used either to inform the development interactions, they also make apparent situations when simi- of practical solutions to problems faced in care delivery, or lar types of interventions would not result in their intended to develop a more conceptual understanding of how to outcomes due to contextual differences triggering alternate organize ideas or frameworks for refining high level stra- mechanisms. RRRs make explicit the amount of literature tegicthinking(Table1). Forthe former,RRRshavebeen readily available for review, and in the case of a limited evi- used to synthesize best practices for specific service dence base, can draw on the input of the expert panel in provisions, such as evidence-based tele-health initiatives understanding the absence of literature and subsequently or an inventory of self-management interventions for developing directions to take in sourcing other relevant lit- chronic disease patients. More conceptual applications have erature for the review. In addition to understanding when produced ‘simple rules’ for guiding large system transform- little is known on a given topic, RRRs also highlight where ation, syntheses of frameworks for supporting evidence- tensions or uncertainties exist in an evidence base, such as informed policy-making and strategies for fostering what the precise characteristics of an effective network organizational partnerships. Methodologically, RRRs have might be for a given purpose within a specific geographic, offered decision makers a tool for merging sources of evi- political and temporal context, or the differential effective- dence including published literature, grey literature and ness of tele-health methods for chronic disease self- expert opinion, and as such, have provided knowledge management for populations within varying contexts. users with the ability to make sense of how and why they The RRRs cited in this analysis have a number of might expect certain outcomes to occur, and the role of strengths. Engaged members of both the expert panel and context in these outcomes. Further, they have helped reference group ensure rigorous methods by providing an knowledge users determine if sufficient substrate exists for extended body of expertise and experience, and validating other structured review exercises. In doing so, RRRs have findings and program theories in an iterative way. In proven useful in providing evidence for and making expli- addition, the participation of the reference group increases cit what is known on a given topic, as well as articulating the local utility of review findings. Moreover, the results where knowledge gaps may exist. speak to the value of diverse data collection methods, From the RRRs completed to date, findings broadly ad- building on the literature through methods such as sur- here to four (often overlapping) classifications: guiding veys and focus groups. In addition, reference groups tend rules for policy making (i.e., evolving patterns of decision- to include (or provide links to) key decision makers in making that can be adapted based on new knowledge commissioning agencies, leading to closer connections be- or contextual factors); knowledge quantification (i.e.,the tween knowledge producers and users. As such, the RRRs amount of literature available that identifies context, mech- completed to date have generally performed well as tools anisms and outcomes for a given topic); understanding for responding to time-sensitive policy issues. tensions/paradoxes in the evidence base, as when similar As with any review process, there are challenges to the changes in context trigger similar mechanisms and produce RRR process. Ensuring that the scope of the review re- divergent outcomes, or when changes in context trigger mains contained and the questions definable can be espe- different mechanisms to produce the same or different out- cially important and difficult in the RRR process. Like all comes; and, reinforcing or refuting beliefs and decisions reviews, difficulties may emerge when limited published or grey literature exists, where the quality of included stud- taken. See Table 2 for case studies that include detail on specific findings for two reviews. RRR provides a tool for ies is poor, or where the findings of studies are heavily developing rules for guiding decision-making, such as the contextualised to particular settings or times. Forming the membership of the reference group and ensuring sustained advantages of investing in both top-down and distributed leadership, as well as on-going performance measurement involvement are challenges for all reviews, particularly in in large system transformations (e.g.,Knowledge AndAc- rapidly changing political environments where member- ship of the reference group may change during the course tion For System Transformation (KAST) review), or the need for innovative inter-organizational investments for of a given project. improving the links between knowledge creation and adop- tion. While these guiding rules can inform the deve- Discussion lopment of strategic directions, they can also be used as The RRR method outlined here and a more traditional tools to confirm or dispel existing beliefs; for example, realist review share numerous similarities (see Table 3). Saul et al. Implementation Science 2013, 8:103 Page 9 of 15 http://www.implementationscience.com/content/8/1/103 Table 2 Case studies Project title: Knowledge And Action For System Transformation (KAST): A Interorganizational Partnerships Systematic Realist Review and Evidence Synthesis of the Role of Government Policy in Directing Large System Transformation Partner: Saskatchewan Ministry of Health National Institutes of Health - Office of Behavioral and Social Science Research and the Canadian Health Services Research Foundation Funder: Canadian Institutes of Health Research Same as partner Length of Six months Two months time: Published Best A, Greenhalgh T, Lewis S, Saul J, Carroll S, Bitz J 2012. Large Riley BL, Best A. (in press). Stakeholders, organizational article: system transformation in health care: A realist systematic review and partnerships, and coalitions. In S. Kahan, A. Gielen, P. Fagan, & evaluation of its usefulness in a policy context. The Milbank L.W. Green (eds). Health Behavior Change in Populations: The Quarterly, 90(3): 421–456. State of the Evidence and Roles for Key Stakeholders. Baltimore, MD: Johns Hopkins University Press. Purpose: The Saskatchewan Ministry of Health has a mandate to support a To provide a RRR of evidence-based principles for effective significant transformation of the provincial health system. Large interorganizational collaboration and partnership, as the system transformation refers to systematic initiatives to create principles relate to two contexts: the collaborations in which coordinated change across organizations working toward shared health research funders engage; and the forms and processes priorities within specified boundaries. The Ministry recognizes that of linkage within universities and their community partners to success hinges on a cultural change based on collaboration, a promote interdisciplinary health research and effective comprehensive innovation strategy for system redesign, and knowledge exchange. The review also summarized the systems integration. Therefore, they seek guidance -- on such existing evidence to identify aspects of interorganizational considerations as successful models and strategies, partnership partnerships (a) where sufficient evidence exists to conduct a principles (including with patients), monitoring and evaluation -- full systematic review, and (b) where insufficient evidence from a systematic review of knowledge on large system exists to articulate the further primary research that is transformation. With all of these considerations, there is an emphasis necessary. on the role of government. Questions 1. What are the key mechanisms or social processes that influence 1. What are the evidence, experiential and theory-based asked in the or drive successful large systems transformation in the health care critical success factors for increased interorganizational review: sector? collaboration? 2. What are the contextual factors that have the most impact 2. What is the nature of the evidence, and what is known (positive or negative) on large systems transformation efforts in the about external validity and generalizability? healthcare sector? 3. Are there identifiable ‘transition’ points in large systems 3. What are the best indicators to monitor collaboration and transformation efforts? If so, how do the key mechanisms and outcomes a) within the research funder context, b) within the contexts interact to produce these changes? university/institutional (researcher performer) context, and c) within the context of the users of research knowledge (research user)? 4. What is the role of government or government-like entities (these 4. What are the strategies most likely to generate buy-in of need definition or at least examples) in large system transformation key influentials in each context for strategic plans and new efforts? initiatives? 5. What does the literature say about the challenges or constraints most likely to affect success in each context? 6. What does the literature say about stages of the development of collaborations and the most effective tactics at each stage? Resulting (In the full report, each has an associated list of contextual factors From a synthesis of literature on the subject, the review evidence and related mechanisms): identified a number of key factors critical to realizing the full statements promise of ‘the collaborative advantage.’ 1. Large system transformation in healthcare systems requires both 1. Clear common aims. It often takes time and cycling top-down leadership that is passionately committed to change, as through the various stages of direction setting, action, and well as distributed leadership and engagement of personnel at all trust building, to develop the overarching partnership-level levels of the system. goal, common language, and aims necessary to enable and sustain a productive partnership. 2. Measurement and reporting on progress toward short- and long- 2. Trust. An essential foundation for any partnership, trust term goals is critical for achieving effective and sustainable large builds on itself over time, often starting with modest, low-risk system transformation. initiatives. 3. Consideration and acknowledgement of historical context will 3. Leadership. Effective interorganizational collaboration help avoid unnecessary pitfalls, and increase buy-in and support requires sustained and engaged leadership, and a shift in from system stakeholders. leadership/management style from ‘command and control’ to facilitating and empowering, and from delegation to participation. Saul et al. Implementation Science 2013, 8:103 Page 10 of 15 http://www.implementationscience.com/content/8/1/103 Table 2 Case studies (Continued) 4. Large system transformation in healthcare systems relies on 4. Sensitivity to power issues. Each partner in a significant physician engagement in the change process. collaborative brings different resources and strengths. Effective collaboration requires acknowledging these differences and taking care in negotiating expectations and the ground rules for decision-making. 5. Large system transformation that aims to increase patient- 5. Membership structures. Shared understandings about centeredness requires significant engagement of patients and what the collaboration involves, and formalized rules, roles families in the change process. and structures enable participation. Both governance and task structures are important. The evidence shows the need for effective coordination infrastructure with agreed-action strategies, and sufficient resources, capacity, and role clarity to support good communication and management functions. Because membership is often dynamic and changing, continuing work is essential to sustain the shared understanding and common focus. Effective coordination structures also help speed the uptake of innovations. 6. Action learning. Effective collaborations continuously improve through feedback loops and reflective shared learning. Strengths of Much of the success of this review was due to the timely topic, This review was positioned to inform a strategic planning the review: which inspired commitment from the reference group and expert initiative for OBSSR; as such it was an important support to panellists. Key to this was the direct involvement of key decision future work for them and was applied in practice almost makers in the Saskatchewan government, which ensured relevancy immediately. This review began with a key set of articles that of the final product and maximized the potential for uptake of the were provided by the research team and expert panellists, findings. The project was designed for and succeeded in blending thus expediting the process. Articles were added as we came practice knowledge with research knowledge. Within the design, across them through citation searching or from the panellists this review was unique in the use of a consensus network to over the duration of the project. As is the case in all validate the findings. This network was made up of national and successful RRRs, the expert panel was engaged and international experts in the area; using an online survey, they were committed to the topic. asked to comment on the five evidence statements and reflect on how well these statements compared with their understanding and experience. Out of 100 people invited, 50 % took part. Considering the voluntary nature and the length of the survey (it took people on average 30 minutes), we deemed this a very good response rate. The consensus network was particularly useful given the literature base was relatively sparse. Also contributing to the success of the project was the opportunity to link the final workshop with another major CIHR initiative, providing an additional opportunity for shared learning and partnering. Challenges As is often a reason for choosing a realist review methodology, the The review had to be done within a two-month timeframe for the amount of published and grey literature on the topic was scarce. with limited resources. This meant no time or money to review: Remedies to this challenge were in-depth and on-going include a key word literature search. However, due to the consultation with a reference group and an expert panel, further research team’s prior work and the expertise from the panel, enhanced by the use of the consensus network. An additional the project was initiated with a solid set of articles for review, challenge was the fact that the findings were intuitive, and did not and completed within the timeframe allocated for the appear to provide ‘novel’ solutions or courses of action to the project. decision makers in the Saskatchewan Ministry of Health. However, they were pleased to have formal confirmation of what they already knew. Considering tacit knowledge often runs ahead of research knowledge, this can be a typical of many reviews, rapid and otherwise. Both are designed to be responsive to local policy needs. process, allowing the entire review process to be com- Both are grounded in realist logic and constructs, expli- pleted within three to six months (see [11] for an ex- citly attempting to understand the interactions between ample). There are limitations associated with reducing the contexts, mechanisms and outcomes. However, they also time available to complete such a review, as are discussed have some key differences, which allow policy makers to below. However, some policy decisions are required in a apply each type of methodology strategically to maximize very short time frame (six months or less), thus making a its utility within a particular local constellation of history, methodology that can gather and synthesize the maximum goals, resources, politics and environment. In particular, amount of evidence in a compressed time period ex- the RRR methodology is explicitly designed to engage tremely useful. knowledge users and review stakeholders to narrowly de- The cost implications for the rapidity of the review fine the research questions, and to streamline the review process are difficult to quantify, since the bulk of our Saul et al. Implementation Science 2013, 8:103 Page 11 of 15 http://www.implementationscience.com/content/8/1/103 Table 3 Comparison of RRR and ‘traditional’ realist review methods RRR method Full realist review method ▪ Examine interactions between mechanisms, context and outcomes ▪ Examine interactions between mechanisms, context and outcomes ▪ Short-term (3- to 6-month turnaround) ▪ Longer term (12- to 24-month turnaround) ▪ Responsive to local policy needs ▪ Reponsive to local policy needs ▪ Results are utilization-focused ▪ Results are utilization-focused ▪ Includes both reference group and expert panel ▪ Likely includes a reference group (or something similar); there is no standard for involvement of an expert panel ▪ Involvement of expert panel allows for conclusions and ▪ Includes both published and grey literature, search process is recommendations to be developed from an emerging/nascent comprehensive (until the point of theory saturation), and program theories literature can be developed over a period of months. work is with RRR, thus prohibiting a good comparison (including those in the process of publication) will be within our own team for rapid vs. traditional realist re- identified and included. Secondly, review of preliminary views. We surmise that the skillset required for the re- findings by the expert panel validates results of the re- view and synthesis increases the cost for personnel, view, ensuring that the interpretation of the literature is although the specific relative costs will vary by project, consistent with the experiences of those currently en- content area, and individuals involved. While the overall gaged in work around the review topic. The intent of in- length of time for RRRs will decrease costs overall, pro- corporating an expert panel is to integrate the lessons ject management costs per month are likely to be greater learned from the literature with those of identified con- for RRRs than for longer realist reviews, given the neces- tent leaders who have experience working to integrate sity to interact frequently with the reference group and knowledge from research, practice and policy perspec- expert panel, and the need to obtain timely responses tives, and to build program theories based on that com- from them. bined knowledge. As discussed, the RRR method outlined here involves The RRR method also differs from a more traditional both a local reference group and an expert panel. A realist review in the role of theory, and specifically the traditional realist review likely engages a group equiva- presentation of theory in the final synthesis product. A lent to a local reference group, but to our knowledge no ‘traditional’ realist review explicitly grounds the review expert panel is engaged as a standard part of the realist process in theory; it produces a narrative summary of approach to knowledge synthesis. Among realist reviews the interactions between context, mechanisms, and out- more generally, ‘advisory groups’ are quite common, and come configurations [8]. The reviewer then constructs often serve the role of a merged ‘reference group’ and one or more ‘middle range theories’ (i.e., ones that ‘in- ‘expert panel’. We have found it useful to separate out volve abstraction… but [are] close enough to observed the ‘expert panel’ into its own functioning unit. To make data to be incorporated in proportions that permit em- the engagement process more efficient, asking the refer- pirical testing’ [19]) to account for the configurations. ence group to focus on the utility of the findings, and The literature is then explored to ‘test’ these theories their relevance to the work at hand within a local con- based on emerging findings from the review. In the RRR text, has worked well. The expert panel is similarly asked method, results are generated through a very similar to focus its efforts on the validity of the findings and our process; however, theory-building is viewed as a support- interpretation of the evidence. The engagement of ex- ive process to identifying families of interventions (I) perts allows for at least two important contributions: and explaining why they would produce outcomes of interest (O) by generating specific changes in context firstly, it provides a validation process for findings that have been generated from an expedited review process. (C) that would then trigger particular mechanisms, ra- In the RRR method, the search strategy is intended to be ther than the primary result. In some cases, however, the content area for the review is nascent enough that there robust, but not comprehensive. Due to the time con- straints, it is fully acknowledged that not all references is insufficient published literature to generate fully devel- related to the research question, or even all that will oped theoretical frameworks in which to ground the re- view. In those cases, the local reference and expert produce theoretical saturation, will be identified and in- cluded in the review. However, by engaging the expert panels assist in producing and refining program theories panel in the searching stage of the project, we can accel- that describe the current best thinking about how pro- grams should work in different contexts. In either case, erate the search process, and at the same time be rea- sonably assured that any key theoretical contributions the focus of the review team is to produce generally ac- within critical articles, reports, or other resources cepted explanations among policy makers and content Saul et al. Implementation Science 2013, 8:103 Page 12 of 15 http://www.implementationscience.com/content/8/1/103 experts for what works within a particular set of parame- available. For example, the evidence informed public ters, rather than aiming to produce explanations that are health policy and practice review consisted primarily of potentially transferrable across contexts and populations. a literature review and synthesis with no reference The distinction may appear to be semantic, but its impact group. On the other end of the spectrum of review com- is very real for both process and product of the review. plexity, the large system transformation review (KAST) included a much broader and more extensive consult- Strengths of the rapid realist review process ation with content experts. While this was not critical to Reviewing the similarities and differences between the the success or usefulness of the final report, it added to RRR and more traditional realist review methods high- the nuanced contextualization provided in the final re- lights several strengths of the RRR approach. Similar to port. Key to the KAST project, the report commissioners realist reviews more generally, the RRR method is time explicitly requested recommendations for government efficient, and may in fact be more amenable to time limi- action that could enhance the likelihood of success for tations than other types of realist reviews (due in part to large system transformation projects. Having such a re- the increased involvement of the expert panel). RRR is quest made it easier to elicit useful feedback from the grounded in local context, with explicit, extensive, iterative content experts that were surveyed for the project. engagement with a local reference group comprising rep- resentatives of potential ‘knowledge users’. The reference Limitations of the rapid realist review process group provides feedback on the project at critical times, While the RRR method has several strengths and advan- including when developing project scope and research tages, as noted above, there are also several limitations questions, identifying key literature, and interpreting re- to the RRR approach. sults (‘Do these findings and recommendations make The RRR method explicitly uses an expedited search sense for our context? Will we be able to use these re- process. This is an advantage with respect to the faster sults?’). As such, the RRR method is responsive to local turnaround time for the finished review, but it may also policy needs and is therefore adherent to the goals of a result in certain resources/references being missed, po- realist approach to knowledge synthesis [3]. As a result, tentially introducing a source of bias. However, the ef- while the RRR method is adaptable to scale, it tends to op- fects of this potential limitation are buffered by the erate most effectively when reviews are commissioned by engagement of the expert panel. Experts validate the those who have authority and power to enable change, findings to ensure that critical elements are not missed, where the policy questions to be answered map clearly and that nuances from emerging practice are included. onto the jurisdiction that is responsible for effecting policy The short time frame in which RRRs are typically change, and where key stakeholders from that jurisdiction conducted can make it difficult to fully theorize the who will be putting policy changes into practice are repre- mechanisms that are identified, as well as the interac- sented on the local reference group (as occurred in the tions between context, mechanisms and outcomes, as is KAST review; see Table 1). When reference groups in- at the core of a full realist review. This may conse- clude (or provide links to) key decision makers in commis- quently limit the generalizability and potency of findings. sioning agencies, this leads to closer connections between Therefore, policy makers interested in sharing results knowledge producers and users. more broadly may need to consider the inclusion of Similar to traditional realist reviews, the RRR method- other examples of when the same (or similar) mechan- ology also works well in situations where there is a small ism is in operation, though this might be an intervention but emerging body of evidence on which to base policy that has not been labelled as being the ‘same’ as the one decisions. The use of the expert panel allows for incorpor- under consideration. However, this is unlikely to be an ation of knowledge-in-the-making into policy recommen- impediment to the actual utilization of the review find- dations, as well as the practice knowledge of those who ings in practice. Working with the local reference group have been engaged in application of research findings to a throughout the project ensures that the project scope, policy context. For example, in the review of methods to research questions, and format of project outcomes will increase organizational capacity to address health literacy meet the needs of those who will use the information. issues within healthcare organizations, the actual pub- Given the time-intensive nature of the RRR review lished literature specifically relating to the review topic process, as well as the critical role of the expert panel, it was limited. As a result, the expert panel helped frame the is unlikely that government agencies or departments will findings significantly to be consistent with what they had have the in-house capacity to complete these reviews. experienced in practice. This enhanced the value of the Adding the RRR method to a policy maker’s toolkit will final report for the knowledge users. routinely require external input from expert teams. Finally, the RRR methodology allows for the ability However, this may actually be an advantage; it allows to scale the project based on the time and resources expert methodologists to be engaged at specific times in Saul et al. Implementation Science 2013, 8:103 Page 13 of 15 http://www.implementationscience.com/content/8/1/103 order to answer specific questions, thus targeting re- use of findings particularly relevant. These threats are sources strategically. One critical advantage is the ex- three fold: periential learning that results, deepening the systems thinking skills required for effective change. The relative 1. Lack of alignment between the policy question and strengths are brought to bear in ways that maximize the the composition/jurisdiction of the reference group. allocation of resources, time and talents, making such a If those engaged as the local reference group aren’t process efficient as well as effective. the ones who will be able to use the results, the Forming the membership of the reference group and process will have significant challenges. ensuring sustained involvement are challenges for all re- 2. Lack of sufficient time to conduct the review, or views, particularly in rapidly changing political environ- barriers that prevent the process from proceeding ments where membership of the reference group may according to schedule. As discussed above, rapid change during the course of a given project. However, feedback from both reference group and expert the role of the reference group in the RRR methodology panel members is necessary for conducting rapid allows key agency or government staff to be engaged in realist reviews. If competing time commitments the process without requiring excessive time commit- prohibit such a level of engagement, the project will ments. Similarly, the engagement of expert panels does lose its ability to produce results in a timely fashion, rely on panellist availability and willingness to contribute the quality of the review may be reduced, or the their time and ideas. Providing honoraria for expert utility of the results may be lessened. panellists, and/or providing opportunities for them to be 3. Lack of continued engagement and evaluation of the involved in dissemination of the work through confer- impact of RRRs. Ideally, RRRs would generate ence presentations, papers, or other avenues, can make evidence-based recommendations for policy action. it easier to engage the right experts at the right time. These actions would be implemented and evaluated to A final potential weakness can result from the require- determine their impact. Additional work needs to be ment that all members of the review team be well versed done to demonstrate and document the impact and in realist review methodology, while at the same time value of the RRR methodology. A lack of engagement commissioners of realist reviews (such as policy makers) or interest in these areas by policy makers will make it do not share the same deep understanding of the meth- more difficult to continue to make the case for the use odology and its value for policy. In our experience, pol- of such methods in the future. icy makers tend to be more concerned about what they are able to accomplish within one specific context (e.g.,a Future work province) rather than adding to the knowledge-base of Future work is needed in several areas to support and middle-range theories. While they need to understand enhance the value of the RRR methodology in deve- C-M-O interactions to be able to identify potentially ef- loping evidence-informed policy. First, as noted above, fective interventions, their focus tends to be more in additional measurement is needed to document the out- intervention (I) – outcome (O) links. We see a growing comes and impact of RRRs. Conducting qualitative re- interest in the how and why interventions work or don’t search and evaluation with the users of products work for various audiences, but understanding the pro- developed using RRR methods will add enormously not gram theories that underlie success or failure of inter- only to the field of implementation science, but will also ventions tends to be lower on the list of priorities than develop a more robust toolkit for policy makers facing producing results. As a result, our reports have had to decisions in a context of uncertainty. Second, and relat- highlight I-O links, and use the program theories we de- edly, work is needed to develop robust criteria with veloped to help explain why those interventions might which the impact of RRR methods can be assessed. work well in a given context for specific populations. Third, a method is needed to explicitly document the boundaries for findings produced with RRR methods. While results of RRRs typically are presented to stake- Potential risks to the impact of rapid realist reviews holders within one specific context, a method is needed The RRR methodology was explicitly developed to bet- for identifying the limits of the review in terms of time ter meet the time-sensitive needs of policy makers for period, geographical area, populations, and environmen- coherent, focused and relevant syntheses of knowledge tal factors that might limit the generalizability of results. to develop evidence-informed policies. There are, how- Finally, a structured program of evaluation needs to be ever, threats to the ability of intended knowledge users developed that guides this process and that is helpful to to put the findings into practice. While these risks are users as well as review methodologists. Currently, all as- applicable to any similar review, the underlying pur- sessments of impact are largely based on user feedback. pose of the RRR methodology makes threats to the Following completing of reviews, a lag-time is required Saul et al. Implementation Science 2013, 8:103 Page 14 of 15 http://www.implementationscience.com/content/8/1/103 to enable actual action on review recommendations. The In sum, the authors may receive financial gain in the future from the publication of this manuscript. In our view, the conflict between academic ideal timeframe, process, and methods for assessing im- and business interest in this area is unavoidable, and bridging the gap pact and outcomes are areas for future research and between these interests is vital to supporting the research to policy and collaboration. practice process. The evidence is clear that knowledge uptake is poor without effective structures to support the process. Conclusions Authors’ contributions Rapid Realist Review offers a useful tool for addressing JS made substantial contributions to: the acquisition, analysis and synthesis of data presented in Table 1; designing, drafting and revising the manuscript. time-sensitive policy decisions. The RRR methodology CW made substantial contributions to: the design, drafting and revision of provides a way of handling the tight time limits and the manuscript; analysis and interpretation of data in Table 1. JB made rapid turnaround necessary that often characterize pol- substantial contributions to: the acquisition, analysis and synthesis of data presented in Table 1; drafting and revising the manuscript. AB made icy decisions. The risks to the rapid nature of the review substantial contributions to: the analysis and synthesis of data presented in methodology are a potential lack of comprehensiveness, Table 1; drafting and revising the manuscript. All authors have approved the or an increased risk of bias. These risks are compensated final version of this manuscript. for by the involvement of content experts who can en- sure that the results are complete, and that they resonate Acknowledgements with practice knowledge. RRR also incorporates local The authors wish to acknowledge Professor Patricia O’Campo and Dr. Geoff Wong for thoughtful (and in the case of Dr. Wong, repeated) reviews of this knowledge users and links to commissioning agencies, manuscript. The authors wish to thank all agencies listed in Table 1 who thus enhancing both usability and utilization of results. have collaborated on previous rapid realist reviews. The authors also wish to Rapid Realist Reviews are scalable with respect to in- thank all associated team members who have contributed to rapid realist reviews outlined in Table 1. volvement of external experts based on the time and re- sources available. The RRR methodology works well in Author details contexts where there is limited evidence due to its in- InSource Research Group, 982 Thermal Dr., Coquitlam, BC V3J 6S1, Canada. North American Research & Analysis, Inc, 1016 11th Avenue NE, Faribault, corporation of practice evidence via expert panels. MN 55021, USA. Centre for Clinical Epidemiology and Evaluation, Vancouver While the end products of RRRs focus less on theory Coastal Health Research Institute, The University of British Columbia, development than on intervention-outcome links, they Research Pavilion, 7th Floor, 828 West 10th Avenue, Vancouver, BC V5Z 1M9, Canada. School of Population Health, The University of Adelaide, Level 7, still rely on program theories to explain how and why 178 North Terrace, Adelaide, SA 5005, Australia. InSource Research Group, certain families of interventions may produce outcomes 6 4581 Anhalt Rd, Kelowna, BC V1W 1P7, Canada. InSource Research Group, of interest within a given context for specific popula- 6975 Marine Drive, West Vancouver, BC V7W 2T4, Canada. School of Population and Public Health, University of British Columbia, 2206 East Mall, tions. It can be more difficult to engage policy makers in Vancouver, BC V6T 1Z3, Canada. theory development, but this does not lessen the utility of realist reviews for producing evidence-informed pol- Received: 18 January 2013 Accepted: 24 August 2013 Published: 5 September 2013 icy. Government agencies interested in using a RRR methodology will likely need to involve expert method- ologists. However, this helps maximize the efficiency of References 1. Lavis J, et al: Towards systematic reviews that inform health care key agency or government staff. management and policy-making. J Health Serv Res Policy 2005, In conclusion, for policy makers faced with making 10(Suppl 1):35–48. difficult decisions in short time frames, related to 2. Pawson R: Evidence-based policy: in search of a method. Evaluation 2002, 8(2):157–181. system-level programs/initiatives/directions, for which 3. Pawson R: Evidence-based policy: a realist perspective. London: Thousand there is sufficient (if limited) published/research and Oaks, Calif.: SAGE; 2006:196. practice-based evidence available, RRR provides a prac- 4. Pawson R: Evidence-based policy: the promise of realist synthesis. Evaluation 2002, 8:340–358. tical, outcomes focused knowledge synthesis method. 5. Green LW: From research to “best practices” in other settings and populations. Am J Health Behav 2001, 25(3):165–178. Competing interests 6. Greenhalgh T, et al: Diffusion of innovations in service organizations: The Rapid Realist Review methodology described in this paper is designed systematic review and recommendations. Milbank Q 2004, 82(4):581–629. to bridge multisectoral barriers to knowledge use. Although the authors 7. Astbury B, Leeuw F: Unpacking black boxes: mechanisms and theory have academic interest in this methodology, they also engage in the building in evaluation. American Journal of Evaluation 2010, 31:363–381. methodology as consultants on a range of projects. Some of these projects 8. Pawson R, et al: Realist review–a new method of systematic review are funded by grants managed by universities, some through contracts to designed for complex policy interventions. J Health Serv Res Policy 2005, the InSource Research Group. InSource was created to provide a vehicle for 10(Suppl 1):21–34. responding to policy maker needs for knowledge synthesis in a more timely 9. Ganann R, Ciliska D, Thomas H: Expediting systematic reviews: methods way than is possible through the normal grant funding process. and implications of rapid reviews. Implement Sci 2010, 5:56. In the interests of transparency and to address conflict of interest concerns, 10. Watt A, et al: Rapid versus full systematic reviews: validity in clinical it is very possible that publication of this paper could enhance InSource’s practice? ANZ J Surg 2008, 78(11):1037–1040. reputation and could result in future contracts for the company. It must also 11. Riley B, Norman C, Best A: Knowledge integration in public health: a rapid be noted that two of the four authors (JB and AB) are InSource directors, review using systems thinking. Evidence and Policy 2012, 8:417–431. and the other two (JS and CW) have worked, or in the future may work, 12. Khangura S, et al: Evidence summaries: the evolution of a rapid review under contract to InSource. approach. Syst Rev 2012, 1(1):10. Saul et al. Implementation Science 2013, 8:103 Page 15 of 15 http://www.implementationscience.com/content/8/1/103 13. Arksey H, O’Malley L: Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, Theory and Practice 2005, 8(1):19–32. 14. Wong G, et al: RAMESES publication standards: realist syntheses. BMC Med 2013, 11:21. 15. Lavis JN, et al: Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med 2012, 9(3):e1001186. 16. Keown K, Van Eerd D, Irvin E: Stakeholder engagement opportunities in systematic reviews: knowledge transfer for policy and practice. J Contin Educ Health Prof 2008, 28(2):67–72. 17. Van de Ven AH, Johnson PE: Knowledge for theory and practice. Acad Manage Rev 2006, 31:802–821. 18. Van de Ven AH: Engaged scholarship: a guide for organizational and social research. Oxford: Oxford University Press; 2007. 19. Merton RK: On theoretical sociology; five essays, old and new. A free press paperback. New York: Free Press; 1967:180. doi:10.1186/1748-5908-8-103 Cite this article as: Saul et al.: A time-responsive tool for informing policy making: rapid realist review. Implementation Science 2013 8:103. Submit your next manuscript to BioMed Central and take full advantage of: • Convenient online submission • Thorough peer review • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit

Journal

Implementation Science – Springer Journals

Published: Sep 5, 2013

There are no references for this article.