Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance

Exploratory studies to decide whether and how to proceed with full-scale evaluations of public... Background: Evaluations of complex interventions in public health are frequently undermined by problems that can be identified before the effectiveness study stage. Exploratory studies, often termed pilot and feasibility studies, are a key step in assessing the feasibility and value of progressing to an effectiveness study. Such studies can provide vital information to support more robust evaluations, thereby reducing costs and minimising potential harms of the intervention. This systematic review forms the first phase of a wider project to address the need for stand-alone guidance for public health researchers on designing and conducting exploratory studies. The review objectives were to identify and examine existing recommendations concerning when such studies should be undertaken, questions they should answer, suitable methods, criteria for deciding whether to progress to an effectiveness study and appropriate reporting. Methods: We searched for published and unpublished guidance reported between January 2000 and November 2016 via bibliographic databases, websites, citation tracking and expert recommendations. Included papers were thematically synthesized. Results: The search retrieved 4095 unique records. Thirty papers were included, representing 25 unique sources of guidance/recommendations. Eight themes were identified: pre-requisites for conducting an exploratory study, nomenclature, guidance for intervention assessment, guidance surrounding any future evaluation study design, flexible versus fixed design, progression criteria to a future evaluation study, stakeholder involvement and reporting of exploratory studies. Exploratory studies were described as being concerned with the intervention content, the future evaluation design or both. However, the nomenclature and endorsed methods underpinning these aims were inconsistent across papers. There was little guidance on what should precede or follow an exploratory study and decision-making surrounding this. Conclusions: Existing recommendations are inconsistent concerning the aims, designs and conduct of exploratory studies, and guidance is lacking on the evidence needed to inform when to proceed to an effectiveness study. Trial registration: PROSPERO 2016, CRD42016047843 Keywords: Public health, Complex interventions, Exploratory studies, Research methods, Study design, Pilot study, Feasibility study * Correspondence: HallingbergBE@cf.ac.uk Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer), Cardiff University, Cardiff, Wales, UK Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 2 of 12 Background remains to be firmly established. Where exploratory Improving public health and disrupting complex prob- studies are conducted poorly, this investment may sim- lems such as smoking, obesity and mental health re- ply lead to expenditure of large amounts of additional quires complex, often multilevel, interventions. Such public money, and several years’ delay in getting evi- interventions are often costly and may cause unantici- dence into the hands of decision-makers, without neces- pated harms and therefore require evaluation using the sarily increasing the likelihood that a future evaluation most robust methods available. However, pressure to will provide useful evidence. identify effective interventions can lead to premature The 2000 MRC guidance used the term ‘exploratory commissioning of large effectiveness studies of poorly trial’ for work conducted prior to a ‘definitive trial’, indi- developed interventions, wasting finite research re- cating that it should primarily address issues concerning sources [1–3]. In the development of pharmaceutical the optimisation, acceptability and delivery of the inter- drugs over 80% fail to reach ‘Phase III’ effectiveness tri- vention [13]. This included adaptation of the interven- als, even after considerable investment [4]. With public tion, consideration of variants of the intervention, health interventions, the historical tendency to rush to testing and refinement of delivery method or content, full evaluation has in some cases led to evaluation fail- assessment of learning curves and implementation strat- ures due to issues which could have been identified at egies and determining the counterfactual. Other possible an earlier stage, such as difficulties recruiting sufficient purposes of exploratory trials included preliminary as- participants [5]. There is growing consensus that im- sessment of effect size in order to calculate the sample proving the effectiveness of public health interventions size for the main trial and other trial design parameters, relies on attention to their design and feasibility [3, 6]. including methods of recruitment, randomisation and However, what constitutes good practice when deciding follow-up. Updated MRC guidance in 2008 moved away when a full evaluation is warranted, what uncertainties from the sole focus on RCTs (randomised controlled tri- should be addressed to inform this decision and how, is als) of its predecessor reflecting recognition that not all unclear. This systematic review aims to synthesize exist- interventions can be tested using an RCT and that the ing sources of guidance for ‘exploratory studies’ which next most robust methods may sometimes be the best we broadly define as studies intended to generate evi- available option [10, 14]. Guidance for exploratory stud- dence needed to decide whether and how to proceed ies prior to a full evaluation have, however, often been with a full-scale effectiveness study. They do this by framed as relevant only where the main evaluation is to optimising or assessing the feasibility of the intervention be an RCT [13, 15]. and/or evaluation design that the effectiveness study However, the goals of exploratory studies advocated by would use. Hence, our definition includes studies vari- research funders have to date varied substantially. For ously referred to throughout the literature as ‘pilot stud- instance, the National Institute for Health Research ies’, ‘feasibility studies’ or ‘exploratory trials’. Our Evaluation Trials and Studies Coordinating Centre definition is consistent with previous work conducted by (NETSCC) definitions of feasibility and pilot studies do Eldridge et al. [7, 8], who define feasibility as an over- not include examination of intervention design, delivery arching concept [8] which assesses; ‘… whether the fu- or acceptability and do not suggest that modifications to ture trial can be done, should be done, and, if so, how’ the intervention prior to full-scale evaluation will arise (p. 2) [7]. However, our definition also includes explora- from these phases. However, the NIHR (National Insti- tory studies to inform non-randomised evaluations, ra- tute of Health Research) portfolio of funded studies indi- ther than a sole focus on trials. cates various uses of terms such as ‘feasibility trial’, ‘pilot The importance of thoroughly establishing the feasibil- trial’ and ‘exploratory trial’ to describe studies with simi- ity of intervention and evaluation plans prior to embark- lar aims, while it is rare for such studies not to include a ing on an expensive, fully powered evaluation was focus on intervention parameters [16–18]. Within the indicated in the Medical Research Council’s (MRC) research literature, there is considerable divergence over framework for the development and evaluation of what exploratory studies should be called, what they complex interventions to improve health [9, 10]. This should achieve, what they should entail, whether and has triggered shifts in the practice of researchers and how they should determine progression to future studies funders toward seeking and granting funding for an ever and how they should be reported [7, 8, 19–21]. growing number of studies to address feasibility issues. This paper presents a systematic review of the existing Such studies are however in themselves often expensive recommendations and guidance on exploratory studies [11, 12]. While there is a compelling case for such relevant to public health, conducted as the first stage of a studies, the extent to which this substantial investment project to develop new MRC guidance on exploratory in exploratory studies has to date improved the effective- studies. This review aims to produce a synthesis of current ness and cost-effectiveness of evidence production guidance/recommendations in relation to the definition, Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 3 of 12 purpose and content of exploratory studies, and what is and backward citation tracking of included papers, as well seen as ‘good’ and ‘bad’ practice as presented by the au- as contacting experts in the field. The first MRC guidance thors. It will provide an overview of key gaps and areas in on developing and evaluating complex interventions in which there is inconsistency within and between docu- health was published in 2000; we therefore excluded guid- ments. The rationale for guidance and recommendations ance published before this year. are presented, as well as the theoretical perspectives informing them. In particular, we examine how far the Selection of included papers existing recommendations answer the following questions: Search results were exported into reference management software Endnote and clearly irrelevant or duplicate re- When is it appropriate to conduct an exploratory study? cords removed by an information specialist. Eligibility cri- What questions should such studies address? teria were applied to abstracts and potentially relevant What are the key methodological considerations in full-text papers by two reviewers working independently answering these questions? in duplicate (BH, JS). Discrepancies were agreed by con- What criteria should inform a decision on whether sensus or by a third reviewer if necessary. Full criteria are to progress to an effectiveness study? shown in Table 1. During screening of eligible studies, it How should exploratory studies be reported? became evident that determining whether or not guidance was applicable to public health was not always clear. The Methods criteria in Table 1 were agreed by the team after a list of This review is reported in accordance with the PRISMA potentially eligible publications were identified. (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement [22]asevidenced in the Quality assessment of included papers PRISMA checklist (see Additional file 1:Table S1). There- Given the nature of publications included (expert guid- view protocol is registered on PROSPERO (registration ance or methodological discussion papers) quality as- number: CRD42016047843; www.crd.york.ac.uk/prospero). sessment was not applicable. Literature search Data extraction and thematic synthesis A comprehensive search (see Additional file 2: A thematic synthesis of guidance within included docu- Appendix) was designed and completed during August to ments was performed [23]. This involved the use of an a November 2016 to identify published and grey literature priori coding framework (based on the projects aims reported between January 2000 and November 2016 that and objectives), developed by RT, JS and DW ([24], see contained guidance and recommendations on exploratory Additional file 2: Appendix). Data were extracted using studies that could have potential relevance to public this schema in qualitative analytic software NVivo by health. Bibliographic databases were CINAHL, Embase, one reviewer (BH). A 10% sample of coded papers was MEDLINE, MEDLINE-In-process, PsycINFO, Web of checked by a second reviewer (JS). Data were then Science and PubMed. Supplementary searches included conceptualised into final themes by agreement (BH, JS, key websites (see Additional file 2: Appendix) and forward DW, RT). Table 1 Eligibility criteria for selecting papers Eligibility criteria Definition of exploratory study � A study which aims to generate the evidence needed to decide whether and how to proceed with a full-scale effectiveness trial, or other study design and are labelled as exploratory/pilot/feasibility/ phase II/proof of concept. Eligible publications may concern some or all of the design features of exploratory studies Nature of guidance � Guidance on the purpose, design, implementation or reporting of exploratory studies and/or � Other publication that reports ‘substantive information’ concerning the conduct of exploratory studies within public health—e.g. worked examples and methodological papers Applicability to public health � Public health audiences clearly among intended users of the guidance (authors are from Public Health departments, cites literature from public health journals, provides public health examples or uses the term ‘public health’ or variants of this, e.g. ‘prevention science’) or � Unspecific audience but of plausible relevance to public health (might, for example, include either an author from a public health research department or a citation to a public health journal) Publication type/source Book, book chapter, journal article, report or readily available doctoral thesis, funding organisation websites (UK and non-UK based) Date and language restrictions Publications reported since 2000 to date (November 2016), in any language. Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 4 of 12 Fig. 1 Flow diagram Results exploratory studies. We categorised these into eight Review statistics themes that capture: pre-requisites for conducting an Four thousand ninety-five unique records were identified exploratory study, nomenclature, guidance for inter- of which 93 were reviewed in full text (see Fig. 1). In total, vention assessment, guidance surrounding the future 30 documents were included in the systematic review evaluation study design, adaptive vs rigid designs, representing 25 unique sets of guidance. Most sources of progression criteria for exploratory studies, stake- guidance did not explicitly identify an intended audience holder involvement and reporting. and guidance varied in its relevance to public health. Table 2 presents an overview of all sources of guidance in- Narrative description of themes cluded in the review with sources of guidance more or less Theme 1: pre-requisites for conducting an exploratory study relevant to public health identified as well as those which Where mentioned, pre-requisite activities included de- specifically applied to exploratory studies with a rando- termining the evidence base, establishing the theoretical mised design. basis for the intervention, identifying the intervention components as well as modelling of the intervention in Findings from guidance order to understand how intervention components Theincludedguidancereportedawide rangeofrecom- interact and impact on final outcomes [9, 25–27]. These mendations on the process of conducting and reporting were often discussed within the context of the MRC’s Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 5 of 12 Table 2 Summary of included guidance Aims outlined and endorsed by authors Intervention future evaluation First author(s) Document Greater relevance Nomenclature Refine Implementation Impact Design Parameter type to public health used feasibility estimation Arain et al. (2010) [19] journal pilot study x publication feasibility study x Bowen et al. (2009) [39] journal ✓ feasibility x x x x publication Campbell et al. (2000) [9] journal ✓ phase II stage; xx x publication exploratory trial Cook et al. (2014) [35] report to pilot study x funder Craig et al. (2008) [10] funder report ✓ feasibility/pilotingxx x stage; pilot study Dixon-Woods et al. (2004) [23] journal exploratory clinical publication trial/phase II Eldridge, Chan et al. (2016) [7] journal ✓ pilot trials xx x publication feasibility trials x x x Eldridge, Lancaster et al. (2016) [8] journal ✓ pilot and feasibility publication studies -feasibility studiesxx that are not pilot studies -non-randomised x pilot studies -randomised pilot x studies Eldridge, Costelloe et al. (2016) [41] journal pilot study x x x x publication Evans et al. (2013) [28] journal ✓ feasibility and publication piloting stage Feeley et al. (2009) [30] journal pilot study x x x x publication Fletcher et al. (2016) [29] journal ✓ pilot study xx x publication feasibility study x x x Hislop et al. (2014) [43] journal pilot study x x publication Lancaster (2004) [36] journal pilot studies x x x publication Lancaster et al. (2015) [20] journal pilot and feasibility xx x publication studies Levati et al. (2016) [31] journal ✓ pilot and feasibility x publication studies Moffatt et al. (2006) [42] journal ✓ pilot study publication (worked example) Möhler et al. (2012) [25] journal feasibility and x publication piloting stage; pilot study Möhler et al. (2013) [26] journal feasibility and x publication piloting stage; pilot study Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 6 of 12 Table 2 Summary of included guidance (Continued) Aims outlined and endorsed by authors Intervention future evaluation First author(s) Document Greater relevance Nomenclature Refine Implementation Impact Design Parameter type to public health used feasibility estimation Möhler et al. (2015) [27] journal feasibility and x publication piloting stage; pilot study Medical Research Council (2000) funder report ✓ phase II stage; xx x x x [13] exploratory trial National Institute for Health funder pilot study xx Research, ‘Feasibility and Pilot document feasibility study x x x studies’ (Accessed 14/10/16) National Institute for Health funder internal pilot x Research, ‘Progression rules for document studies internal pilot studies for HTA trials’ (Accessed 14/10/16) National Institute for Health funder website pilot study xx Research, ‘Glossary’ (14/10/16) feasibility study x x O'Cathain et al. (2015) [15] journal ✓ feasibility study x x x x publication Shanyinde et al. (2011) [21] journal pilot / feasibility publication trials Strong et al. (2009) [40] journal ✓ pilot intervention x x publication Taylor et al. (2015) [34] book chapter ✓ pilot study xx feasibility study x x Westlund et al. (2016) [38] journal pilot study x x publication Wight et al. (2015) [6] journal ✓ specific term not xx x x x publication stated Guidance with greater relevance to public health included those where public health audiences was clearly among intended users of the guidance (authors are from Public Health departments, cites literature from public health journals, provides public health examples or uses the term ‘public health’ or variants of this, e.g. ‘prevention science’, ‘health improvement’). Guidance with less relevance was not specific about the intended audience but was of plausible relevance to public health (might, for example, include either an author from a public health research department or a citation to a public health journal). b c Authors make distinctions between the terms “pilot study” and “feasibility study”. Aims of exploratory studies presented in the table map onto aims presented in themes 3 (Guidance for intervention assessment) and 4 (Guidance surrounding the future evaluation design) intervention development-evaluation cycle [6, 9, 10, 13, Table 3 Frequency of nomenclature used 25–28]. Understanding how intervention components Nomenclature Number of sources interact with various contextual settings [6, 27, 29] and Pilot trial/study 16 identifying unintended harms [6, 29] as well as potential Feasibility trial/study 8 implementation issues [6, 9, 10, 30] were also Feasibility and piloting stage 5 highlighted. There was an absence of detail in judging Pilot and/or feasibility trial/study 5 when these above conditions were met sufficiently for Phase II trial/study 3 moving onto an exploratory study. Exploratory trial/study 3 Theme 2: nomenclature Other terms (external pilot, feasibility Terms presented once or studies but not pilot studies, non- twice across different A wide range of terms were used, sometimes inter- randomised pilot studies, randomised sources of guidance. changeably, to describe exploratory studies with the feasibility studies, randomised pilot most common being pilot trial/study. Table 3 shows the studies, exploratory pilot study, feasibility RCT, formative study, frequency of the terms used in guidance including other phase II stage, pilot RCT, process terms endorsed. evaluation with a pilot trial, Different terminology did not appear to be consistently randomised feasibility trial, randomised pilot trial) associated with specific study purposes (see theme 3), as illustrated in Table 2. ‘Pilot’ and ‘feasibility’ studies were Note: terms are not mutually exclusive Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 7 of 12 sometimes used interchangeably [10, 20, 25–28, 31]while Intervention implementation others made distinctions between the two according to There was agreement across a wide range of guidance design features or particular aims [7, 8, 19, 29, 32–34]. For that exploratory studies could explore key uncertainties example, some described pilot studies as a smaller version related to intervention implementation, such as accept- of a future RCT to run in miniature [7, 8, 19, 29, 32–34] ability, feasibility or practicality. Notably these terms and was sometimes associated with a randomised design were often ill-defined and used interchangeably. Accept- [32, 34], but not always [7, 8]. In contrast, feasibility ability was considered in terms of recipients’ reactions studies were used as an umbrella term by Eldridge et al. [7, 8, 29, 32, 39] while others were also attentive to feasi- with pilot studies representing a subset of feasibility bility from the perspective of intervention providers, de- studies [7, 8]: ‘We suggest that researchers view feasibility liverers and health professionals [6, 9, 29, 30, 34, 39]. as an overarching concept, with all studies done in prepar- Implementation, feasibility, fidelity and ‘practicality’ ex- ation for a main study open to being called feasibility stud- plored the likelihood of being able to deliver in practice ies, and with pilot studies as a subset of feasibility studies.’ what was intended [25–27, 30, 39]. These were some- (p. 18) [8]. times referred to as aims within an embedded process Feasibility studies could focus on particular interven- evaluation that took place alongside an exploratory tion and trial design elements [29, 32] which may not in- study, although the term process evaluation was never clude randomisation [32, 34]. Internal pilot studies were defined [7, 10, 15, 29, 40]. primarily viewed as part of the full trial [8, 32, 35–38] Qualitative research was encouraged for assessment of and are therefore not depicted under nomenclature intervention acceptability [21] or for implementation in Table 3. (e.g. via non-participant observation [15]). Caution was While no sources explicitly stated that an exploratory recommended with regards to focus groups where there is study should focus on one area and not the other, aims a risk of masking divergent views [15]. Others recom- and associated methods of exploratory studies diverged mended quantitative surveys to examine retention rates into two separate themes. They pertained to either and reasons for dropout [7, 30]. Furthermore, several examining the intervention itself or the future evaluation sources emphasised the importance of testing implemen- design, and are detailed below in themes 3 and 4. tation in a range of contexts [15, 29, 39, 41]—especially in less socioeconomically advantaged groups, to examine the Theme 3: guidance for intervention assessment risk of widening health inequalities [29, 39]. Sources of guidance endorsed exploratory studies having One source of guidance considered whether randomisa- formative purposes (i.e. refining the intervention and tion was required for assessing intervention acceptability, addressing uncertainties related to intervention imple- believing this to be unnecessary but also suggesting it mentation [13, 15, 29, 31, 39]) as well as summative could ‘potentially depend on preference among interven- goals (i.e. assessing the potential impact of an interven- tions offered in the main trial’ ([21]; p. 9). Thus, issues of tion or its promise [6, 13, 39]). intervention acceptability, particularly within multi-arm trials, may relate to clinical equipoise and acceptability of Refining the intervention and underlying theory randomisation procedures among participants [30]. Some guidance suggested that changes could be made within exploratory studies to refine the intervention and Appropriateness of assessing intervention impact underlying theory [15, 29, 31] and adapt intervention Several sources of guidance discussed the need to under- content to a new setting [39]. However, guidance was stand the impact of the intervention, including harms, not clear on what constituted minor vs. major changes benefits or unintended consequences [6, 7, 15, 29, 39]. and implications for progression criteria (see theme 6). Much of the guidance focused on statistical tests of ef- When making changes to the intervention or underlying fectiveness with disagreement on the soundness of this theory, some guidance recommended this take place aim, although qualitative methods were also recom- during the course of the exploratory study (see theme 5). mended [15, 42]. Some condemned statistically testing Others highlighted the role of using a multi-arm design for effectiveness [7, 20, 29, 32, 41], as such studies are to select the contents of the intervention before a full often underpowered, hence leading to imprecise and po- evaluation [13] and to assess potential mechanisms of tentially misleading estimates of effect sizes [7, 20]. multiple different interventions or intervention compo- Others argued that an estimate of likely effect size could nents [29]. Several sources highlighted the role of quali- evidence the intervention was working as intended and tative research in optimising or refining an intervention, not having serious unintended harms [6] and thus be particularly for understanding the components of the used to calculate the power for the full trial [13]. Later logic model [29] and surfacing hidden aspects of the guidance from the MRC is more ambiguous than earlier intervention important for delivering outcomes [15]. guidance, stating that estimates should be interpreted Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 8 of 12 with caution, while simultaneously stating ‘safe’ assump- participants are assisted in changing their behaviour; al- tions of effect sizes as a pre-requisite before continuing though assessing single blinding may be possible [30]. to a full evaluation [10]. NIHR guidance, which distin- Qualitative [15, 30, 34], quantitative [34] and mixed guished between pilot and feasibility studies, supported methods [7] were endorsed for assessing these processes. the assessment of a primary outcome in pilot studies, al- Reflecting the tendency for guidance of exploratory stud- though it is unclear whether this is suggesting that a ies to be limited to studies in preparation for RCTs, dis- pilot should involve an initial test of changes in the pri- cussion of the role of randomisation at the exploratory mary outcome, or simply that the primary outcome study stage featured heavily in guidance. Randomisation should be measured in the same way as it would be in a within an exploratory study was considered necessary full evaluation. By contrast, for ‘feasibility studies’, it indi- for examining feasibility of recruitment, consent to ran- cated that an aim may include designing an outcome domisation, retention, contamination or maintenance of measure to be used in a full evaluation. blinding in the control and intervention groups, ran- Others made the case for identifying evidence of domisation procedures and whether all the components potential effectiveness, including use of interim or surro- of a protocol can work together, although randomisation gate endpoints [7, 41], defined as ‘…variables on the was not deemed necessary to assess outcome burden causal pathway of what might eventually be the primary and participant eligibility [21, 30, 34]. While there was outcome in the future definitive RCT, or outcomes at consensus about what issues could be assessed through early time points, in order to assess the potential for the randomisation, sources disagreed on whether random- intervention to affect likely outcomes in the future de- isation should always precede a future evaluation study, finitive RCT…’ [7] (p. 14). even if that future study is to be an RCT. Contention Randomisation was implied as a design feature of ex- seemed to be linked to variation in nomenclature and ploratory studies when estimating an effect size estimate associated aims. For example, some defined pilot study of the intervention as it maximised the likelihood that ob- as a study run in miniature to test how all its compo- served differences are due to intervention [9, 39], with nents work together, thereby dictating a randomised de- guidance mostly written from a starting assumption that sign [32, 34]. Yet for feasibility studies, randomisation full evaluation will take the form of an RCT and guidance was only necessary if it reduced the uncertainties in esti- focused less on exploratory studies for quasi-experimental mating parameters for the future evaluation [32, 34]. or other designs. For studies that aim to assess potential Similarly, other guidance highlighted an exploratory effectiveness using a surrogate or interim outcome, using study (irrespective of nomenclature) should address the a standard sample size calculation was recommended to main uncertainties, and thus may not depend on ran- ensure adequate power, although it was noted that this domisation [8, 15]. aim is rare in exploratory studies [7]. Estimating parameters of the future evaluation design Theme 4: guidance surrounding the future evaluation A number of sources recommended exploratory studies design should inform the parameters of the future evaluation Sources consistently advocated assessing the feasibility design. Areas for investigation included estimating sam- of study procedures or estimating parameters of the fu- ple sizes required for the future evaluation (e.g. measur- ture evaluation. Recommendations are detailed below. ing outcomes [32, 35]; power calculations [13]; derive effect size estimates [6, 7, 39]; estimating target differ- Assessing feasibility of the future evaluation design ences [35, 43]; deciding what outcomes to measure and Assessing feasibility of future evaluation procedures was how [9, 20, 30, 36]; assessing quality of measures (e.g. commonly recommended [6, 7, 10, 15, 30, 32–34, 37, for reliability/ validity/ feasibility/ sensitivity [7, 20, 30]; 41] to avert problems that could undermine the conduct identification of control group [9, 13]; recruitment, con- or acceptability of future evaluation [6, 15, 30]. A wide sent and retention rates [10, 13, 20, 30, 32, 34, 36]; range of procedures were suggested as requiring assess- and information on the cost of the future evaluation ments of feasibility including data collection [20, 30, 34, design [9, 30, 36]. 36, 41], participant retention strategies [13], randomisa- While qualitative methods were deemed useful for tion [7, 13, 20, 30, 34, 36, 38, 41], recruitment methods selecting outcomes and their suitable measures [15], [13, 30, 32, 34, 35, 38, 41], running the full trial protocol most guidance concentrated on quantitative methods for [20, 30, 36], the willingness of participants to be rando- estimating future evaluation sample sizes. This was con- mised [30, 32] and issues of contamination [30]. There tentious due to the potential to over- or under- was disagreement concerning the appropriateness of estimate sample sizes required in a future evaluation assessing blinding in exploratory studies [7, 30, 34], with due to the lack of precision of estimates from a small one source noting double blinding is difficult when pilot [20, 30, 41]. Estimating sample sizes from effect Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 9 of 12 size estimates in an exploratory study was nevertheless ar- others highlight specific items concerning cost- gued by some to be useful if there was scant literature and effectiveness [10], refining causal hypotheses to be tested the exploratory study used the same design and outcome in a future evaluation [29] and meeting recruitment tar- as the future evaluation [30, 39]. Cluster RCTs, which are gets [20, 34]. As discussed in themes 3 and 4, statistically common in public health interventions, were specifically testing for effectiveness and using effect sizes for power earmarked as unsuitable for estimating parameters for calculations was cautioned by some, and so criteria sample size calculations (e.g. intra-cluster correlation coef- based on effect sizes were not specified [38]. ficients) as well as recruitment and follow-up rates with- Greater discussion was devoted to how to weight evi- out additional information from other resources, because dence from an exploratory study that addressed multiple a large number of clusters and individual participants aims and used different methods. Some explicitly stated would be required [41]. Others referred to ‘rules of thumb’ progression criteria should not be judged as strict when determining sample sizes in an exploratory study thresholds but as guidelines using, for example, a traffic with numbers varying between 10 and 75 participants per lights system with varying levels of acceptability [7, 41]. trial arm in individually randomised studies [7, 30, 36]. Others highlighted a realist approach, moving away Several also recommended the need to consider a desired from binary indicators to focusing on ‘what is feasible meaningful difference in the health outcomes from a and acceptable for whom and under what circum- future evaluation and the appropriate sample size stances’ [29]. In light of the difficulties surrounding needed to detect this, rather than conducting sample interpretation of effect estimates, several sources rec- size calculations using estimates of likely effect size ommended qualitative findings from exploratory stud- from pilot data [30, 35, 38, 43]. ies should be more influential than quantitative A randomised design was deemed unnecessary for es- findings [15, 38]. timating costs or selecting outcomes, although was val- Interestingly, there was ambiguity regarding progres- ued for estimating recruitment and retention rates for sion when exploratory findings indicated substantial intervention and control groups [21, 34]. Where guid- changes to the intervention or evaluation design. ance indicated the estimation of an effect size appropri- Sources considering this issue suggested that if ‘exten- ate to inform the sample size for a future evaluation, a sive changes’ or ‘major modifications’ are made to either randomised design was deemed necessary [9, 39]. (note they did not specify what qualified as such), re- searchers should return to the exploratory [21, 30]or Theme 5: flexible vs. fixed design intervention development phases [15]. Sources stated that exploratory studies could employ a ‘Alternatively, at the feasibility phase, researchers may rigid or flexible design. With the latter, the design can identify fundamental problems with the intervention or change during the course of the study, which is useful trial conduct and return to the development phase ra- for making changes to the intervention, as well as the fu- ther than proceed to a full trial.’ (p. 1) [15]. ture evaluation design [6, 13, 15, 31]. Here, qualitative As described previously, however, the threshold at data can be analysed as it is collected, shaping the ex- which changes are determined to be ‘major’ remained ploratory study process, for instance sampling of subse- ambiguous. While updated MRC guidance [10] moved quent data collection points [15], and clarifying to a more iterative model, accepting that movement implications for intervention effectiveness [31]. back between feasibility/piloting and intervention devel- In contrast, fixed exploratory studies were encouraged opment may sometimes be needed, there was no guid- when primarily investigating the future evaluation param- ance on under what conditions movement between eters and processes [13]. It may be that the nomenclature these two stages should take place. used in some guidance (e.g. pilot studies that are described as miniature versions of the evaluation) is suggesting a Theme 7: stakeholder involvement distinction between more flexible vs. more stringent de- Several sources recommended a range of stakeholders signs. In some guidance, it was not mentioned whether (e.g. intervention providers, intervention recipients, pub- changes should be made during the course of an explora- lic representatives as well as practitioners who might use tory study or afterwards, in order to get the best possible the evidence produced by the full trial) be involved in design for the future evaluation [6, 7, 21]. the planning and running of the exploratory study to ensure exploratory studies reflect the realities of inter- Theme 6: progression criteria to a future evaluation study vention setting [15, 28, 31, 32, 39, 40]. In particular, Little guidance was provided on what should be consid- community-based participatory approaches were recom- ered when formulating progression criteria for continu- mended [15, 39]. While many highlighted the value of ing onto a future evaluation study. Some focussed on stakeholders on Trial Steering Committees and other the relevant uncertainties of feasibility [32, 39], while similar study groups [15, 28, 40], some warned about Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 10 of 12 equipoise between researchers and stakeholders [15, 40] substantial that movement back toward intervention de- and also cautioned against researchers conflating velopment rather than forward to a full evaluation re- stakeholder involvement with qualitative research [15]. mains ambiguous. Consistent with past reviews which ‘Although patient and public representatives on research adopted a narrower focus on studies with randomised teams can provide helpful feedback on the intervention, designs [21] or in preparation for a randomised trial [8, this does not constitute qualitative research and may not 36] and limited searches of guidance in medical journals result in sufficiently robust data to inform the appropriate [19, 36], terms to describe exploratory studies were in- development of the intervention.’ (p. 8) [15]. consistent, with a distinction sometimes made between pilot and feasibility studies, though with others using Theme 8: reporting of exploratory studies these terms interchangeably. Detailed recommendations for reporting exploratory The review identifies a number of key areas of dis- studies were recently provided in new Consolidated agreement or limited guidance in regards to the critical Standards of Reporting Trials (CONSORT) guidance by aims of exploratory studies and addressing uncertainties Eldridge et al. [7]. In addition to this, recurrent points which might undermine a future evaluation, and how were brought up by other sources of guidance. Most these aims should be achieved. There was much dis- notably, it was recommended exploratory studies be agreement for example on whether exploratory studies published in peer-reviewed journals as this can provide should include a preliminary assessment of intervention useful information to other researchers on what has effects to inform decisions on progression to a full been done, what did not work and what might be most evaluation, and the appropriateness of using estimates of appropriate [15, 30]. An exploratory study may also re- effect from underpowered data (from non-representative sult in multiple publications, but should provide refer- samples and a study based on a not fully optimised ver- ence to other work carried out in the same exploratory sion of the intervention) to power a future evaluation study [7, 15]. Several sources of guidance also highlight study. Most guidance focused purely on studies in prep- that exploratory studies should be appropriately labelled aration for RCTs; nevertheless, guidance varied on in the title/abstract to enable easy identification; how- whether randomisation was a necessary feature of the ever, the nomenclature suggested varied depending on exploratory study, even where a future evaluation study guidance [7, 8, 15]. was an RCT. Guidance was often difficult to assess re- garding its applicability to public health research, with Discussion many sources focusing on literature and practice primar- While exploratory studies—carried out to inform ily from clinical research, and limited consideration of decisions about whether and how to proceed with an ef- the transferability of these problems and proposed solu- fectiveness study [7, 8]—are increasingly recognised as im- tions to complex social interventions, such as those in portant in the efficient evaluation of complex public public health. Progression criteria were highlighted as health interventions, our findings suggest that this area re- important by some as a means of preventing biased post mains in need of consistent standards to inform practice. hoc cases for continuation. However, there was a lack of At present, there are multiple definitions of exploratory guidance on how to devise progression criteria and pro- studies, a lack of consensus on a number of key issues, cesses for assessing whether these had been sufficiently and a paucity of detailed guidance on how to approach met. Where they had not been met, there was a lack of the main uncertainties such studies aim to address prior guidance on how to decide whether the exploratory to proceeding to a full evaluation. study had generated sufficient insight about uncertain- Existing guidance commonly focuses almost exclu- ties that the expense of a further feasibility study would sively on testing methodological parameters [33], such not be justified prior to large-scale evaluation. as recruitment and retention, although in practice, it is Although our review included a broad focus on guid- unusual for such studies not to also focus on the feasi- ance of exploratory studies from published and grey lit- bility of the intervention itself. Where intervention feasi- erature and moved beyond a focus on studies conducted bility is discussed, there is limited guidance on when an in preparation for an RCT specifically, a number of limi- intervention is ‘ready’ for an exploratory study and a lack tations should be noted. Guidance from other areas of of demarcation between intervention development and social intervention research where challenges may be pre-evaluation work to understand feasibility. Some similar to those in public health (e.g. education, social guidance recognised that an intervention continues to work and business) may not have been captured by our develop throughout an exploratory study, with distinc- search strategy. We found few worked examples of ex- tions made between ‘optimisation/refinement’ (i.e. minor ploratory studies in public health that provided substan- refinements to the intervention) vs. ‘major changes’. tial information from learned experience and practice. However, the point at which changes become so Hence, the review drew largely on recommendations Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 11 of 12 from funding organisations, or relatively abstract guid- Acknowledgements We thank the Specialist Unit for Review Evidence (SURE) at Cardiff University, ance from teams of researchers, with fewer clear exam- including Mala Mann, Helen Morgan, Alison Weightman and Lydia Searchfield, ples of how these recommendations are grounded in for their assistance with developing and conducting the literature search. experience from the conduct of such studies. As such, it Funding should be acknowledged that these documents represent This study is supported by funding from the Methodology Research Panel one element within a complex system of research pro- (MR/N015843/1). LM, SS and DW are supported by the UK Medical Research duction and may not necessarily fully reflect what is tak- Council (MC_UU_12017/14) and the Chief Scientist Office (SPHSU14). PC is supported by the UK Medical Research Council (MC_UU_12017/15) and the ing place in the conduct of exploratory studies. Finally, Chief Scientist Office (SPHSU15). The work was also undertaken with the treating sources of guidance as independent from each support of The Centre for the Development and Evaluation of Complex other does not reflect how some recommendations Interventions for Public Health Improvement (DECIPHer), a UKCRC Public Health Research Centre of Excellence. Joint funding (MR/KO232331/1) from developed over time (see for example [7, 8, 20, 36, 41]). the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, the Welsh Government and the Wellcome Trust, under the auspices of the UK Clinical Research Conclusion Collaboration, is gratefully acknowledged. There is inconsistent guidance, and for some key issues Availability of data and materials a lack of guidance, for exploratory studies of complex The datasets generated and/or analysed during the current study are not public health interventions. As this lack of guidance for publicly available due to copyright infringement. researchers in public health continues, the implications Authors’ contributions and consequences could be far reaching. It is unclear LM, GM, PC, MR, JS, RT and SS were involved in the development of the how researchers use existing guidance to shape decision- study. RT, JS, DW and BH were responsible for the data collection, overseen making in the conduct of exploratory studies, and in by LM and GM. Data analysis was undertaken by BH guided by RT, JS, DW and GM. The manuscript was prepared by BH, RT, DW, JS and GM. All doing so, how they adjudicate between various conflict- authors contributed to the final version of the manuscript. LM is the ing perspectives. This systematic review has aimed principal investigator with overall responsibility for the project. GM is Cardiff largely to identify areas of agreement and disagreement lead for the project. All authors read and approved the final manuscript. as a starting point in bringing order to this somewhat Ethics approval and consent to participate chaotic field of work. Following this systematic review, Not applicable. our next step is to conduct an audit of published public Competing interests health exploratory studies in peer-reviewed journals, to The authors declare that they have no competing interests. assess current practice and how this reflects the reviewed guidance. As part of a wider study, funded by Publisher’sNote the MRC/NIHR Methodology Research Programme to Springer Nature remains neutral with regard to jurisdictional claims in develop GUidance for Exploratory STudies of complex published maps and institutional affiliations. public health interventions (GUEST; Moore L, et al. Author details Exploratory studies to inform full scale evaluations of Centre for the Development and Evaluation of Complex Interventions for complex public health interventions: the need for guid- Public Health Improvement (DECIPHer), Cardiff University, Cardiff, Wales, UK. 2 3 Centre for Trials Research, Cardiff University, Cardiff, Wales, UK. MRC/CSO ance, submitted), the review has informed a Delphi sur- Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. vey of researchers, funders and publishers of public Specialist Unit for Review Evidence, Cardiff University, Cardiff, Wales, UK. health research. In turn, this will contribute to a consen- Received: 6 February 2018 Accepted: 7 May 2018 sus meeting which aims to reach greater unanimity on the aims of exploratory studies, and how these can most efficiently address uncertainties which may undermine a References 1. Kessler R, Glasgow RE. A proposal to speed translation of healthcare full-scale evaluation. research into practice: dramatic change is needed. Am J Prev Med. 2011;40:637–44. 2. Sanson-Fisher RW, Bonevski B, Green LW, D’Este C. Limitations of the Additional files randomized controlled trial in evaluating population-based health interventions. Am J Prev Med. 2007;33:155–61. Additional file 1: Table S1. PRISMA checklist. (DOC 62 kb) 3. Speller V, Learmonth A, Harrison D. The search for evidence of effective Additional file 2: Appendix 1. Search strategies and websites. health promotion. BMJ. 1997;315(7104):361. Appendix 2. Coding framework. (DOCX 28 kb) 4. Arrowsmith J, Miller P. Trial watch: phase II failures: 2008–2010. Nat Rev Drug Discov. 2011;10(5):328–9. 5. National Institute for Health Research. Weight loss maintenance in adults Abbreviations (WILMA). https://www.journalslibrary.nihr.ac.uk/programmes/hta/084404/#/. CONSORT: Consolidated Standards of Reporting Trials; GUEST: GUidance for Accessed 13 Dec 2017. Exploratory STudies of complex public health interventions; MRC: Medical 6. Wight D, Wimbush E, Jepson R, Doi L. Six steps in quality intervention Research Council; NETSCC: National Institute of Health Research Evaluation development (6SQuID). J Epidemiol Community Health. 2015;70:520–5. Trials and Studies Coordinating Centre; NIHR: National Institute for Health 7. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. Research; PRISMA: Preferred Reporting Items for Systematic Reviews and CONSORT 2010 statement: extension to randomised pilot and feasibility Meta-Analyses; RCT: Randomised controlled trial trials. Pilot Feasibility Stud. 2016;2:64. Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 12 of 12 8. Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman 29. Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex CL, et al. Defining feasibility and pilot studies in preparation for randomised intervention science: applying realist principles across all phases of the controlled trials: development of a conceptual framework. PLoS One. 2016; Medical Research Council framework for developing and evaluating 11:e0150205. complex interventions. Evaluation. 2016;22:286–303. 9. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL. Framework for design 30. Feeley N, Cossette S, Côté J, Héon M, Stremler R, Martorella G, et al. The and evaluation of complex interventions to improve health. BMJ. 2000; importance of piloting an RCT intervention. CJNR. 2009;41:84–99. 321(7262):694. 31. Levati S, Campbell P, Frost R, Dougall N, Wells M, Donaldson C, et al. Optimisation of complex health interventions prior to a randomised controlled 10. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. trial: a scoping review of strategies used. Pilot Feasibility Stud. 2016;2:1. Developing and evaluating complex interventions: new guidance. Medical 32. National Institute for Health Research. Feasibility and pilot studies. Available Research Council. 2008; from: http://www.nihr.ac.uk/CCF/RfPB/FAQs/Feasibility_and_pilot_studies. 11. National Institute for Health Research. The Filter FE Challenge: pilot trial and pdf. Accessed 14 Oct 2016. process evaluation of a multi-level smoking prevention intervention in 33. National Institute for Health Research. Glossary | Pilot studies 2015 http://www. further education settings. Available from: https://www.journalslibrary.nihr. nets.nihr.ac.uk/glossary?result_1655_result_page=P.Accessed 14 Oct 2016. ac.uk/programmes/phr/134202/#/. Accessed 25 Jan 2018. 34. Taylor RS, Ukoumunne OC, Warren FC. How to use feasibility and pilot trials 12. National Institute for Health Research. Adapting and piloting the ASSIST to test alternative methodologies and methodological procedures proir to model of informal peer-led intervention delivery to the Talk to Frank drug full-scale trials. In: Richards DA, Hallberg IR, editors. Complex interventions in prevention programme in UK secondary schools (ASSIST+Frank): an health: an overview of research methods. New York: Routledge; 2015. exploratory trial. https://www.journalslibrary.nihr.ac.uk/programmes/phr/ 35. Cook JA, Hislop J, Adewuyi TE, Harrild K, Altman DG, Ramsay CR et al. 12306003/#/. Accessed 25 Jan 2018. Assessing methods to specify the target difference for a randomised 13. Medical Research Council. A framework for the development and evaluation controlled trial: DELTA (Difference ELicitation in TriAls) review. Health of RCTs for complex interventions to improve health. London: Medical Technology Assessment (Winchester, England). 2014;18:v–vi. Research Council; 2000. 36. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: 14. Bonell CP, Hargreaves JR, Cousens SN, Ross DA, Hayes R, Petticrew M, et al. recommendations for good practice. J Eval Clin Pract. 2004;10:307–12. Alternatives to randomisation in the evaluation of public-health 37. National Institute for Health Research. Progression rules for internal pilot studies interventions: design challenges and solutions. J Epidemiol Community for HTA trials [14/10/2016]. Available from: http://www.nets.nihr.ac.uk/__data/ Health. 2009; https://doi.org/10.1136/jech.2008.082602. assets/pdf_file/0018/115623/Progression_rules_for_internal_pilot_studies.pdf. 15. O’Cathain A, Hoddinott P, Lewin S, Thomas KJ, Young B, Adamson J, et al. 38. Westlund E, Stuart EA. The nonuse, misuse, and proper use of pilot studies Maximising the impact of qualitative research in feasibility studies for in experimental evaluation research. Am J Eval. 2016;2:246–61. randomised controlled trials: guidance for researchers. Pilot Feasibility 39. Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. Stud. 2015;1(1):32. How we design feasibility studies. Am J Prev Med. 2009;36:452–7. 16. National Institute for Health Research. An exploratory trial to evaluate the 40. Strong LL, Israel BA, Schulz AJ, Reyes A, Rowe Z, Weir SS et al. Piloting effects of a physical activity intervention as a smoking cessation induction interventions within a community-based participatory research framework: and cessation aid among the ‘hard to reach’. https://www.journalslibrary. lessons learned from the healthy environments partnership. Prog nihr.ac.uk/programmes/hta/077802/#/. Accessed 13 Dec 2017. Community Health Partnersh. 2009;3:327–34. 17. National Institue for Health Research. Initiating change locally in bullying 41. Eldridge SM, Costelloe CE, Kahan BC, Lancaster GA, Kerry SM. How big and aggression through the school environment (INCLUSIVE): pilot should the pilot study for my cluster randomised trial be? Stat Methods randomised controlled trial. https://www.journalslibrary.nihr.ac.uk/hta/ Med Res. 2016;25:1039–56. hta19530/#/abstract. Accessed 13 Dec 2017. 42. Moffatt S, White M, Mackintosh J, Howel D. Using quantitative and 18. National Institute for Health Resarch. Increasing boys' and girls' intention to qualitative data in health services research—what happens when mixed avoid teenage pregnancy: a cluster randomised control feasibility trial of an method findings conflict? [ISRCTN61522618]. BMC Health Serv Res. 2006;6:1. interactive video drama based intervention in post-primary schools in 43. Hislop J, Adewuyi TE, Vale LD, Harrild K, Fraser C, Gurung T et al. Methods Northern Ireland. https://www.journalslibrary.nihr.ac.uk/phr/phr05010/ for specifying the target difference in a randomised controlled trial: the #/abstract. Accessed 13 Dec 2017. Difference ELicitation in TriAls (DELTA) systematic review. PLoS Med. 2014; 19. Arain M, Campbell, MJ, Cooper CL, Lancaster GA. What is a pilot or 11:e1001645. feasibility study? A review of current practice and editorial policy BMC Med Res Methodol. 2010;10:67. 20. Lancaster GA. Pilot and feasibility studies come of age! Pilot Feasibility Stud. 2015;1:1. 21. Shanyinde M, Pickering RM, Weatherall M. Questions asked and answered in pilot and feasibility randomized controlled trials. BMC Med Res Methodol. 2011;11:117. 22. Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;151:e1000097. 23. Dixon-Woods M, Agarwal S, Jones D. Intergrative approaches to qualitative and quantitative evidence. London: Health Development Agency; 2004. 24. Ritchie J, Spencer L, O’Connor W. Carrying out qualitative analysis. Qualitative research practice: a guide for social science students and researchers. 2003;1. 25. Möhler R, Bartoszek G, Köpke S, Meyer G. Proposed criteria for reporting the development and evaluation of complex interventions in healthcare (CReDECI): guideline development. IJNS. 2012;49(1):40–6. 26. Möhler R, Bartoszek G, Meyer G. Quality of reporting of complex healthcare interventions and applicability of the CReDECI list—a survey of publications indexed in PubMed. BMC Med Res Methodol. 2013;13:1. 27. Möhler R, Köpke S, Meyer G. Criteria for reporting the development and evaluation of complex interventions in healthcare: revised guideline (CReDECI 2). Trials. 2015;16(204):1. 28. Evans BA, Bedson E, Bell P, Hutchings H, Lowes L, Rea D, et al. Involving service users in trials: developing a standard operating procedure. Trials. 2013;14(1):1. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Pilot and Feasibility Studies Springer Journals

Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance

Free
12 pages

Loading next page...
 
/lp/springer_journal/exploratory-studies-to-decide-whether-and-how-to-proceed-with-full-GvsHUNGhsB
Publisher
Springer Journals
Copyright
Copyright © 2018 by The Author(s).
Subject
Medicine & Public Health; Medicine/Public Health, general; Biomedicine, general; Statistics for Life Sciences, Medicine, Health Sciences
eISSN
2055-5784
D.O.I.
10.1186/s40814-018-0290-8
Publisher site
See Article on Publisher Site

Abstract

Background: Evaluations of complex interventions in public health are frequently undermined by problems that can be identified before the effectiveness study stage. Exploratory studies, often termed pilot and feasibility studies, are a key step in assessing the feasibility and value of progressing to an effectiveness study. Such studies can provide vital information to support more robust evaluations, thereby reducing costs and minimising potential harms of the intervention. This systematic review forms the first phase of a wider project to address the need for stand-alone guidance for public health researchers on designing and conducting exploratory studies. The review objectives were to identify and examine existing recommendations concerning when such studies should be undertaken, questions they should answer, suitable methods, criteria for deciding whether to progress to an effectiveness study and appropriate reporting. Methods: We searched for published and unpublished guidance reported between January 2000 and November 2016 via bibliographic databases, websites, citation tracking and expert recommendations. Included papers were thematically synthesized. Results: The search retrieved 4095 unique records. Thirty papers were included, representing 25 unique sources of guidance/recommendations. Eight themes were identified: pre-requisites for conducting an exploratory study, nomenclature, guidance for intervention assessment, guidance surrounding any future evaluation study design, flexible versus fixed design, progression criteria to a future evaluation study, stakeholder involvement and reporting of exploratory studies. Exploratory studies were described as being concerned with the intervention content, the future evaluation design or both. However, the nomenclature and endorsed methods underpinning these aims were inconsistent across papers. There was little guidance on what should precede or follow an exploratory study and decision-making surrounding this. Conclusions: Existing recommendations are inconsistent concerning the aims, designs and conduct of exploratory studies, and guidance is lacking on the evidence needed to inform when to proceed to an effectiveness study. Trial registration: PROSPERO 2016, CRD42016047843 Keywords: Public health, Complex interventions, Exploratory studies, Research methods, Study design, Pilot study, Feasibility study * Correspondence: HallingbergBE@cf.ac.uk Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer), Cardiff University, Cardiff, Wales, UK Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 2 of 12 Background remains to be firmly established. Where exploratory Improving public health and disrupting complex prob- studies are conducted poorly, this investment may sim- lems such as smoking, obesity and mental health re- ply lead to expenditure of large amounts of additional quires complex, often multilevel, interventions. Such public money, and several years’ delay in getting evi- interventions are often costly and may cause unantici- dence into the hands of decision-makers, without neces- pated harms and therefore require evaluation using the sarily increasing the likelihood that a future evaluation most robust methods available. However, pressure to will provide useful evidence. identify effective interventions can lead to premature The 2000 MRC guidance used the term ‘exploratory commissioning of large effectiveness studies of poorly trial’ for work conducted prior to a ‘definitive trial’, indi- developed interventions, wasting finite research re- cating that it should primarily address issues concerning sources [1–3]. In the development of pharmaceutical the optimisation, acceptability and delivery of the inter- drugs over 80% fail to reach ‘Phase III’ effectiveness tri- vention [13]. This included adaptation of the interven- als, even after considerable investment [4]. With public tion, consideration of variants of the intervention, health interventions, the historical tendency to rush to testing and refinement of delivery method or content, full evaluation has in some cases led to evaluation fail- assessment of learning curves and implementation strat- ures due to issues which could have been identified at egies and determining the counterfactual. Other possible an earlier stage, such as difficulties recruiting sufficient purposes of exploratory trials included preliminary as- participants [5]. There is growing consensus that im- sessment of effect size in order to calculate the sample proving the effectiveness of public health interventions size for the main trial and other trial design parameters, relies on attention to their design and feasibility [3, 6]. including methods of recruitment, randomisation and However, what constitutes good practice when deciding follow-up. Updated MRC guidance in 2008 moved away when a full evaluation is warranted, what uncertainties from the sole focus on RCTs (randomised controlled tri- should be addressed to inform this decision and how, is als) of its predecessor reflecting recognition that not all unclear. This systematic review aims to synthesize exist- interventions can be tested using an RCT and that the ing sources of guidance for ‘exploratory studies’ which next most robust methods may sometimes be the best we broadly define as studies intended to generate evi- available option [10, 14]. Guidance for exploratory stud- dence needed to decide whether and how to proceed ies prior to a full evaluation have, however, often been with a full-scale effectiveness study. They do this by framed as relevant only where the main evaluation is to optimising or assessing the feasibility of the intervention be an RCT [13, 15]. and/or evaluation design that the effectiveness study However, the goals of exploratory studies advocated by would use. Hence, our definition includes studies vari- research funders have to date varied substantially. For ously referred to throughout the literature as ‘pilot stud- instance, the National Institute for Health Research ies’, ‘feasibility studies’ or ‘exploratory trials’. Our Evaluation Trials and Studies Coordinating Centre definition is consistent with previous work conducted by (NETSCC) definitions of feasibility and pilot studies do Eldridge et al. [7, 8], who define feasibility as an over- not include examination of intervention design, delivery arching concept [8] which assesses; ‘… whether the fu- or acceptability and do not suggest that modifications to ture trial can be done, should be done, and, if so, how’ the intervention prior to full-scale evaluation will arise (p. 2) [7]. However, our definition also includes explora- from these phases. However, the NIHR (National Insti- tory studies to inform non-randomised evaluations, ra- tute of Health Research) portfolio of funded studies indi- ther than a sole focus on trials. cates various uses of terms such as ‘feasibility trial’, ‘pilot The importance of thoroughly establishing the feasibil- trial’ and ‘exploratory trial’ to describe studies with simi- ity of intervention and evaluation plans prior to embark- lar aims, while it is rare for such studies not to include a ing on an expensive, fully powered evaluation was focus on intervention parameters [16–18]. Within the indicated in the Medical Research Council’s (MRC) research literature, there is considerable divergence over framework for the development and evaluation of what exploratory studies should be called, what they complex interventions to improve health [9, 10]. This should achieve, what they should entail, whether and has triggered shifts in the practice of researchers and how they should determine progression to future studies funders toward seeking and granting funding for an ever and how they should be reported [7, 8, 19–21]. growing number of studies to address feasibility issues. This paper presents a systematic review of the existing Such studies are however in themselves often expensive recommendations and guidance on exploratory studies [11, 12]. While there is a compelling case for such relevant to public health, conducted as the first stage of a studies, the extent to which this substantial investment project to develop new MRC guidance on exploratory in exploratory studies has to date improved the effective- studies. This review aims to produce a synthesis of current ness and cost-effectiveness of evidence production guidance/recommendations in relation to the definition, Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 3 of 12 purpose and content of exploratory studies, and what is and backward citation tracking of included papers, as well seen as ‘good’ and ‘bad’ practice as presented by the au- as contacting experts in the field. The first MRC guidance thors. It will provide an overview of key gaps and areas in on developing and evaluating complex interventions in which there is inconsistency within and between docu- health was published in 2000; we therefore excluded guid- ments. The rationale for guidance and recommendations ance published before this year. are presented, as well as the theoretical perspectives informing them. In particular, we examine how far the Selection of included papers existing recommendations answer the following questions: Search results were exported into reference management software Endnote and clearly irrelevant or duplicate re- When is it appropriate to conduct an exploratory study? cords removed by an information specialist. Eligibility cri- What questions should such studies address? teria were applied to abstracts and potentially relevant What are the key methodological considerations in full-text papers by two reviewers working independently answering these questions? in duplicate (BH, JS). Discrepancies were agreed by con- What criteria should inform a decision on whether sensus or by a third reviewer if necessary. Full criteria are to progress to an effectiveness study? shown in Table 1. During screening of eligible studies, it How should exploratory studies be reported? became evident that determining whether or not guidance was applicable to public health was not always clear. The Methods criteria in Table 1 were agreed by the team after a list of This review is reported in accordance with the PRISMA potentially eligible publications were identified. (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement [22]asevidenced in the Quality assessment of included papers PRISMA checklist (see Additional file 1:Table S1). There- Given the nature of publications included (expert guid- view protocol is registered on PROSPERO (registration ance or methodological discussion papers) quality as- number: CRD42016047843; www.crd.york.ac.uk/prospero). sessment was not applicable. Literature search Data extraction and thematic synthesis A comprehensive search (see Additional file 2: A thematic synthesis of guidance within included docu- Appendix) was designed and completed during August to ments was performed [23]. This involved the use of an a November 2016 to identify published and grey literature priori coding framework (based on the projects aims reported between January 2000 and November 2016 that and objectives), developed by RT, JS and DW ([24], see contained guidance and recommendations on exploratory Additional file 2: Appendix). Data were extracted using studies that could have potential relevance to public this schema in qualitative analytic software NVivo by health. Bibliographic databases were CINAHL, Embase, one reviewer (BH). A 10% sample of coded papers was MEDLINE, MEDLINE-In-process, PsycINFO, Web of checked by a second reviewer (JS). Data were then Science and PubMed. Supplementary searches included conceptualised into final themes by agreement (BH, JS, key websites (see Additional file 2: Appendix) and forward DW, RT). Table 1 Eligibility criteria for selecting papers Eligibility criteria Definition of exploratory study � A study which aims to generate the evidence needed to decide whether and how to proceed with a full-scale effectiveness trial, or other study design and are labelled as exploratory/pilot/feasibility/ phase II/proof of concept. Eligible publications may concern some or all of the design features of exploratory studies Nature of guidance � Guidance on the purpose, design, implementation or reporting of exploratory studies and/or � Other publication that reports ‘substantive information’ concerning the conduct of exploratory studies within public health—e.g. worked examples and methodological papers Applicability to public health � Public health audiences clearly among intended users of the guidance (authors are from Public Health departments, cites literature from public health journals, provides public health examples or uses the term ‘public health’ or variants of this, e.g. ‘prevention science’) or � Unspecific audience but of plausible relevance to public health (might, for example, include either an author from a public health research department or a citation to a public health journal) Publication type/source Book, book chapter, journal article, report or readily available doctoral thesis, funding organisation websites (UK and non-UK based) Date and language restrictions Publications reported since 2000 to date (November 2016), in any language. Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 4 of 12 Fig. 1 Flow diagram Results exploratory studies. We categorised these into eight Review statistics themes that capture: pre-requisites for conducting an Four thousand ninety-five unique records were identified exploratory study, nomenclature, guidance for inter- of which 93 were reviewed in full text (see Fig. 1). In total, vention assessment, guidance surrounding the future 30 documents were included in the systematic review evaluation study design, adaptive vs rigid designs, representing 25 unique sets of guidance. Most sources of progression criteria for exploratory studies, stake- guidance did not explicitly identify an intended audience holder involvement and reporting. and guidance varied in its relevance to public health. Table 2 presents an overview of all sources of guidance in- Narrative description of themes cluded in the review with sources of guidance more or less Theme 1: pre-requisites for conducting an exploratory study relevant to public health identified as well as those which Where mentioned, pre-requisite activities included de- specifically applied to exploratory studies with a rando- termining the evidence base, establishing the theoretical mised design. basis for the intervention, identifying the intervention components as well as modelling of the intervention in Findings from guidance order to understand how intervention components Theincludedguidancereportedawide rangeofrecom- interact and impact on final outcomes [9, 25–27]. These mendations on the process of conducting and reporting were often discussed within the context of the MRC’s Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 5 of 12 Table 2 Summary of included guidance Aims outlined and endorsed by authors Intervention future evaluation First author(s) Document Greater relevance Nomenclature Refine Implementation Impact Design Parameter type to public health used feasibility estimation Arain et al. (2010) [19] journal pilot study x publication feasibility study x Bowen et al. (2009) [39] journal ✓ feasibility x x x x publication Campbell et al. (2000) [9] journal ✓ phase II stage; xx x publication exploratory trial Cook et al. (2014) [35] report to pilot study x funder Craig et al. (2008) [10] funder report ✓ feasibility/pilotingxx x stage; pilot study Dixon-Woods et al. (2004) [23] journal exploratory clinical publication trial/phase II Eldridge, Chan et al. (2016) [7] journal ✓ pilot trials xx x publication feasibility trials x x x Eldridge, Lancaster et al. (2016) [8] journal ✓ pilot and feasibility publication studies -feasibility studiesxx that are not pilot studies -non-randomised x pilot studies -randomised pilot x studies Eldridge, Costelloe et al. (2016) [41] journal pilot study x x x x publication Evans et al. (2013) [28] journal ✓ feasibility and publication piloting stage Feeley et al. (2009) [30] journal pilot study x x x x publication Fletcher et al. (2016) [29] journal ✓ pilot study xx x publication feasibility study x x x Hislop et al. (2014) [43] journal pilot study x x publication Lancaster (2004) [36] journal pilot studies x x x publication Lancaster et al. (2015) [20] journal pilot and feasibility xx x publication studies Levati et al. (2016) [31] journal ✓ pilot and feasibility x publication studies Moffatt et al. (2006) [42] journal ✓ pilot study publication (worked example) Möhler et al. (2012) [25] journal feasibility and x publication piloting stage; pilot study Möhler et al. (2013) [26] journal feasibility and x publication piloting stage; pilot study Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 6 of 12 Table 2 Summary of included guidance (Continued) Aims outlined and endorsed by authors Intervention future evaluation First author(s) Document Greater relevance Nomenclature Refine Implementation Impact Design Parameter type to public health used feasibility estimation Möhler et al. (2015) [27] journal feasibility and x publication piloting stage; pilot study Medical Research Council (2000) funder report ✓ phase II stage; xx x x x [13] exploratory trial National Institute for Health funder pilot study xx Research, ‘Feasibility and Pilot document feasibility study x x x studies’ (Accessed 14/10/16) National Institute for Health funder internal pilot x Research, ‘Progression rules for document studies internal pilot studies for HTA trials’ (Accessed 14/10/16) National Institute for Health funder website pilot study xx Research, ‘Glossary’ (14/10/16) feasibility study x x O'Cathain et al. (2015) [15] journal ✓ feasibility study x x x x publication Shanyinde et al. (2011) [21] journal pilot / feasibility publication trials Strong et al. (2009) [40] journal ✓ pilot intervention x x publication Taylor et al. (2015) [34] book chapter ✓ pilot study xx feasibility study x x Westlund et al. (2016) [38] journal pilot study x x publication Wight et al. (2015) [6] journal ✓ specific term not xx x x x publication stated Guidance with greater relevance to public health included those where public health audiences was clearly among intended users of the guidance (authors are from Public Health departments, cites literature from public health journals, provides public health examples or uses the term ‘public health’ or variants of this, e.g. ‘prevention science’, ‘health improvement’). Guidance with less relevance was not specific about the intended audience but was of plausible relevance to public health (might, for example, include either an author from a public health research department or a citation to a public health journal). b c Authors make distinctions between the terms “pilot study” and “feasibility study”. Aims of exploratory studies presented in the table map onto aims presented in themes 3 (Guidance for intervention assessment) and 4 (Guidance surrounding the future evaluation design) intervention development-evaluation cycle [6, 9, 10, 13, Table 3 Frequency of nomenclature used 25–28]. Understanding how intervention components Nomenclature Number of sources interact with various contextual settings [6, 27, 29] and Pilot trial/study 16 identifying unintended harms [6, 29] as well as potential Feasibility trial/study 8 implementation issues [6, 9, 10, 30] were also Feasibility and piloting stage 5 highlighted. There was an absence of detail in judging Pilot and/or feasibility trial/study 5 when these above conditions were met sufficiently for Phase II trial/study 3 moving onto an exploratory study. Exploratory trial/study 3 Theme 2: nomenclature Other terms (external pilot, feasibility Terms presented once or studies but not pilot studies, non- twice across different A wide range of terms were used, sometimes inter- randomised pilot studies, randomised sources of guidance. changeably, to describe exploratory studies with the feasibility studies, randomised pilot most common being pilot trial/study. Table 3 shows the studies, exploratory pilot study, feasibility RCT, formative study, frequency of the terms used in guidance including other phase II stage, pilot RCT, process terms endorsed. evaluation with a pilot trial, Different terminology did not appear to be consistently randomised feasibility trial, randomised pilot trial) associated with specific study purposes (see theme 3), as illustrated in Table 2. ‘Pilot’ and ‘feasibility’ studies were Note: terms are not mutually exclusive Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 7 of 12 sometimes used interchangeably [10, 20, 25–28, 31]while Intervention implementation others made distinctions between the two according to There was agreement across a wide range of guidance design features or particular aims [7, 8, 19, 29, 32–34]. For that exploratory studies could explore key uncertainties example, some described pilot studies as a smaller version related to intervention implementation, such as accept- of a future RCT to run in miniature [7, 8, 19, 29, 32–34] ability, feasibility or practicality. Notably these terms and was sometimes associated with a randomised design were often ill-defined and used interchangeably. Accept- [32, 34], but not always [7, 8]. In contrast, feasibility ability was considered in terms of recipients’ reactions studies were used as an umbrella term by Eldridge et al. [7, 8, 29, 32, 39] while others were also attentive to feasi- with pilot studies representing a subset of feasibility bility from the perspective of intervention providers, de- studies [7, 8]: ‘We suggest that researchers view feasibility liverers and health professionals [6, 9, 29, 30, 34, 39]. as an overarching concept, with all studies done in prepar- Implementation, feasibility, fidelity and ‘practicality’ ex- ation for a main study open to being called feasibility stud- plored the likelihood of being able to deliver in practice ies, and with pilot studies as a subset of feasibility studies.’ what was intended [25–27, 30, 39]. These were some- (p. 18) [8]. times referred to as aims within an embedded process Feasibility studies could focus on particular interven- evaluation that took place alongside an exploratory tion and trial design elements [29, 32] which may not in- study, although the term process evaluation was never clude randomisation [32, 34]. Internal pilot studies were defined [7, 10, 15, 29, 40]. primarily viewed as part of the full trial [8, 32, 35–38] Qualitative research was encouraged for assessment of and are therefore not depicted under nomenclature intervention acceptability [21] or for implementation in Table 3. (e.g. via non-participant observation [15]). Caution was While no sources explicitly stated that an exploratory recommended with regards to focus groups where there is study should focus on one area and not the other, aims a risk of masking divergent views [15]. Others recom- and associated methods of exploratory studies diverged mended quantitative surveys to examine retention rates into two separate themes. They pertained to either and reasons for dropout [7, 30]. Furthermore, several examining the intervention itself or the future evaluation sources emphasised the importance of testing implemen- design, and are detailed below in themes 3 and 4. tation in a range of contexts [15, 29, 39, 41]—especially in less socioeconomically advantaged groups, to examine the Theme 3: guidance for intervention assessment risk of widening health inequalities [29, 39]. Sources of guidance endorsed exploratory studies having One source of guidance considered whether randomisa- formative purposes (i.e. refining the intervention and tion was required for assessing intervention acceptability, addressing uncertainties related to intervention imple- believing this to be unnecessary but also suggesting it mentation [13, 15, 29, 31, 39]) as well as summative could ‘potentially depend on preference among interven- goals (i.e. assessing the potential impact of an interven- tions offered in the main trial’ ([21]; p. 9). Thus, issues of tion or its promise [6, 13, 39]). intervention acceptability, particularly within multi-arm trials, may relate to clinical equipoise and acceptability of Refining the intervention and underlying theory randomisation procedures among participants [30]. Some guidance suggested that changes could be made within exploratory studies to refine the intervention and Appropriateness of assessing intervention impact underlying theory [15, 29, 31] and adapt intervention Several sources of guidance discussed the need to under- content to a new setting [39]. However, guidance was stand the impact of the intervention, including harms, not clear on what constituted minor vs. major changes benefits or unintended consequences [6, 7, 15, 29, 39]. and implications for progression criteria (see theme 6). Much of the guidance focused on statistical tests of ef- When making changes to the intervention or underlying fectiveness with disagreement on the soundness of this theory, some guidance recommended this take place aim, although qualitative methods were also recom- during the course of the exploratory study (see theme 5). mended [15, 42]. Some condemned statistically testing Others highlighted the role of using a multi-arm design for effectiveness [7, 20, 29, 32, 41], as such studies are to select the contents of the intervention before a full often underpowered, hence leading to imprecise and po- evaluation [13] and to assess potential mechanisms of tentially misleading estimates of effect sizes [7, 20]. multiple different interventions or intervention compo- Others argued that an estimate of likely effect size could nents [29]. Several sources highlighted the role of quali- evidence the intervention was working as intended and tative research in optimising or refining an intervention, not having serious unintended harms [6] and thus be particularly for understanding the components of the used to calculate the power for the full trial [13]. Later logic model [29] and surfacing hidden aspects of the guidance from the MRC is more ambiguous than earlier intervention important for delivering outcomes [15]. guidance, stating that estimates should be interpreted Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 8 of 12 with caution, while simultaneously stating ‘safe’ assump- participants are assisted in changing their behaviour; al- tions of effect sizes as a pre-requisite before continuing though assessing single blinding may be possible [30]. to a full evaluation [10]. NIHR guidance, which distin- Qualitative [15, 30, 34], quantitative [34] and mixed guished between pilot and feasibility studies, supported methods [7] were endorsed for assessing these processes. the assessment of a primary outcome in pilot studies, al- Reflecting the tendency for guidance of exploratory stud- though it is unclear whether this is suggesting that a ies to be limited to studies in preparation for RCTs, dis- pilot should involve an initial test of changes in the pri- cussion of the role of randomisation at the exploratory mary outcome, or simply that the primary outcome study stage featured heavily in guidance. Randomisation should be measured in the same way as it would be in a within an exploratory study was considered necessary full evaluation. By contrast, for ‘feasibility studies’, it indi- for examining feasibility of recruitment, consent to ran- cated that an aim may include designing an outcome domisation, retention, contamination or maintenance of measure to be used in a full evaluation. blinding in the control and intervention groups, ran- Others made the case for identifying evidence of domisation procedures and whether all the components potential effectiveness, including use of interim or surro- of a protocol can work together, although randomisation gate endpoints [7, 41], defined as ‘…variables on the was not deemed necessary to assess outcome burden causal pathway of what might eventually be the primary and participant eligibility [21, 30, 34]. While there was outcome in the future definitive RCT, or outcomes at consensus about what issues could be assessed through early time points, in order to assess the potential for the randomisation, sources disagreed on whether random- intervention to affect likely outcomes in the future de- isation should always precede a future evaluation study, finitive RCT…’ [7] (p. 14). even if that future study is to be an RCT. Contention Randomisation was implied as a design feature of ex- seemed to be linked to variation in nomenclature and ploratory studies when estimating an effect size estimate associated aims. For example, some defined pilot study of the intervention as it maximised the likelihood that ob- as a study run in miniature to test how all its compo- served differences are due to intervention [9, 39], with nents work together, thereby dictating a randomised de- guidance mostly written from a starting assumption that sign [32, 34]. Yet for feasibility studies, randomisation full evaluation will take the form of an RCT and guidance was only necessary if it reduced the uncertainties in esti- focused less on exploratory studies for quasi-experimental mating parameters for the future evaluation [32, 34]. or other designs. For studies that aim to assess potential Similarly, other guidance highlighted an exploratory effectiveness using a surrogate or interim outcome, using study (irrespective of nomenclature) should address the a standard sample size calculation was recommended to main uncertainties, and thus may not depend on ran- ensure adequate power, although it was noted that this domisation [8, 15]. aim is rare in exploratory studies [7]. Estimating parameters of the future evaluation design Theme 4: guidance surrounding the future evaluation A number of sources recommended exploratory studies design should inform the parameters of the future evaluation Sources consistently advocated assessing the feasibility design. Areas for investigation included estimating sam- of study procedures or estimating parameters of the fu- ple sizes required for the future evaluation (e.g. measur- ture evaluation. Recommendations are detailed below. ing outcomes [32, 35]; power calculations [13]; derive effect size estimates [6, 7, 39]; estimating target differ- Assessing feasibility of the future evaluation design ences [35, 43]; deciding what outcomes to measure and Assessing feasibility of future evaluation procedures was how [9, 20, 30, 36]; assessing quality of measures (e.g. commonly recommended [6, 7, 10, 15, 30, 32–34, 37, for reliability/ validity/ feasibility/ sensitivity [7, 20, 30]; 41] to avert problems that could undermine the conduct identification of control group [9, 13]; recruitment, con- or acceptability of future evaluation [6, 15, 30]. A wide sent and retention rates [10, 13, 20, 30, 32, 34, 36]; range of procedures were suggested as requiring assess- and information on the cost of the future evaluation ments of feasibility including data collection [20, 30, 34, design [9, 30, 36]. 36, 41], participant retention strategies [13], randomisa- While qualitative methods were deemed useful for tion [7, 13, 20, 30, 34, 36, 38, 41], recruitment methods selecting outcomes and their suitable measures [15], [13, 30, 32, 34, 35, 38, 41], running the full trial protocol most guidance concentrated on quantitative methods for [20, 30, 36], the willingness of participants to be rando- estimating future evaluation sample sizes. This was con- mised [30, 32] and issues of contamination [30]. There tentious due to the potential to over- or under- was disagreement concerning the appropriateness of estimate sample sizes required in a future evaluation assessing blinding in exploratory studies [7, 30, 34], with due to the lack of precision of estimates from a small one source noting double blinding is difficult when pilot [20, 30, 41]. Estimating sample sizes from effect Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 9 of 12 size estimates in an exploratory study was nevertheless ar- others highlight specific items concerning cost- gued by some to be useful if there was scant literature and effectiveness [10], refining causal hypotheses to be tested the exploratory study used the same design and outcome in a future evaluation [29] and meeting recruitment tar- as the future evaluation [30, 39]. Cluster RCTs, which are gets [20, 34]. As discussed in themes 3 and 4, statistically common in public health interventions, were specifically testing for effectiveness and using effect sizes for power earmarked as unsuitable for estimating parameters for calculations was cautioned by some, and so criteria sample size calculations (e.g. intra-cluster correlation coef- based on effect sizes were not specified [38]. ficients) as well as recruitment and follow-up rates with- Greater discussion was devoted to how to weight evi- out additional information from other resources, because dence from an exploratory study that addressed multiple a large number of clusters and individual participants aims and used different methods. Some explicitly stated would be required [41]. Others referred to ‘rules of thumb’ progression criteria should not be judged as strict when determining sample sizes in an exploratory study thresholds but as guidelines using, for example, a traffic with numbers varying between 10 and 75 participants per lights system with varying levels of acceptability [7, 41]. trial arm in individually randomised studies [7, 30, 36]. Others highlighted a realist approach, moving away Several also recommended the need to consider a desired from binary indicators to focusing on ‘what is feasible meaningful difference in the health outcomes from a and acceptable for whom and under what circum- future evaluation and the appropriate sample size stances’ [29]. In light of the difficulties surrounding needed to detect this, rather than conducting sample interpretation of effect estimates, several sources rec- size calculations using estimates of likely effect size ommended qualitative findings from exploratory stud- from pilot data [30, 35, 38, 43]. ies should be more influential than quantitative A randomised design was deemed unnecessary for es- findings [15, 38]. timating costs or selecting outcomes, although was val- Interestingly, there was ambiguity regarding progres- ued for estimating recruitment and retention rates for sion when exploratory findings indicated substantial intervention and control groups [21, 34]. Where guid- changes to the intervention or evaluation design. ance indicated the estimation of an effect size appropri- Sources considering this issue suggested that if ‘exten- ate to inform the sample size for a future evaluation, a sive changes’ or ‘major modifications’ are made to either randomised design was deemed necessary [9, 39]. (note they did not specify what qualified as such), re- searchers should return to the exploratory [21, 30]or Theme 5: flexible vs. fixed design intervention development phases [15]. Sources stated that exploratory studies could employ a ‘Alternatively, at the feasibility phase, researchers may rigid or flexible design. With the latter, the design can identify fundamental problems with the intervention or change during the course of the study, which is useful trial conduct and return to the development phase ra- for making changes to the intervention, as well as the fu- ther than proceed to a full trial.’ (p. 1) [15]. ture evaluation design [6, 13, 15, 31]. Here, qualitative As described previously, however, the threshold at data can be analysed as it is collected, shaping the ex- which changes are determined to be ‘major’ remained ploratory study process, for instance sampling of subse- ambiguous. While updated MRC guidance [10] moved quent data collection points [15], and clarifying to a more iterative model, accepting that movement implications for intervention effectiveness [31]. back between feasibility/piloting and intervention devel- In contrast, fixed exploratory studies were encouraged opment may sometimes be needed, there was no guid- when primarily investigating the future evaluation param- ance on under what conditions movement between eters and processes [13]. It may be that the nomenclature these two stages should take place. used in some guidance (e.g. pilot studies that are described as miniature versions of the evaluation) is suggesting a Theme 7: stakeholder involvement distinction between more flexible vs. more stringent de- Several sources recommended a range of stakeholders signs. In some guidance, it was not mentioned whether (e.g. intervention providers, intervention recipients, pub- changes should be made during the course of an explora- lic representatives as well as practitioners who might use tory study or afterwards, in order to get the best possible the evidence produced by the full trial) be involved in design for the future evaluation [6, 7, 21]. the planning and running of the exploratory study to ensure exploratory studies reflect the realities of inter- Theme 6: progression criteria to a future evaluation study vention setting [15, 28, 31, 32, 39, 40]. In particular, Little guidance was provided on what should be consid- community-based participatory approaches were recom- ered when formulating progression criteria for continu- mended [15, 39]. While many highlighted the value of ing onto a future evaluation study. Some focussed on stakeholders on Trial Steering Committees and other the relevant uncertainties of feasibility [32, 39], while similar study groups [15, 28, 40], some warned about Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 10 of 12 equipoise between researchers and stakeholders [15, 40] substantial that movement back toward intervention de- and also cautioned against researchers conflating velopment rather than forward to a full evaluation re- stakeholder involvement with qualitative research [15]. mains ambiguous. Consistent with past reviews which ‘Although patient and public representatives on research adopted a narrower focus on studies with randomised teams can provide helpful feedback on the intervention, designs [21] or in preparation for a randomised trial [8, this does not constitute qualitative research and may not 36] and limited searches of guidance in medical journals result in sufficiently robust data to inform the appropriate [19, 36], terms to describe exploratory studies were in- development of the intervention.’ (p. 8) [15]. consistent, with a distinction sometimes made between pilot and feasibility studies, though with others using Theme 8: reporting of exploratory studies these terms interchangeably. Detailed recommendations for reporting exploratory The review identifies a number of key areas of dis- studies were recently provided in new Consolidated agreement or limited guidance in regards to the critical Standards of Reporting Trials (CONSORT) guidance by aims of exploratory studies and addressing uncertainties Eldridge et al. [7]. In addition to this, recurrent points which might undermine a future evaluation, and how were brought up by other sources of guidance. Most these aims should be achieved. There was much dis- notably, it was recommended exploratory studies be agreement for example on whether exploratory studies published in peer-reviewed journals as this can provide should include a preliminary assessment of intervention useful information to other researchers on what has effects to inform decisions on progression to a full been done, what did not work and what might be most evaluation, and the appropriateness of using estimates of appropriate [15, 30]. An exploratory study may also re- effect from underpowered data (from non-representative sult in multiple publications, but should provide refer- samples and a study based on a not fully optimised ver- ence to other work carried out in the same exploratory sion of the intervention) to power a future evaluation study [7, 15]. Several sources of guidance also highlight study. Most guidance focused purely on studies in prep- that exploratory studies should be appropriately labelled aration for RCTs; nevertheless, guidance varied on in the title/abstract to enable easy identification; how- whether randomisation was a necessary feature of the ever, the nomenclature suggested varied depending on exploratory study, even where a future evaluation study guidance [7, 8, 15]. was an RCT. Guidance was often difficult to assess re- garding its applicability to public health research, with Discussion many sources focusing on literature and practice primar- While exploratory studies—carried out to inform ily from clinical research, and limited consideration of decisions about whether and how to proceed with an ef- the transferability of these problems and proposed solu- fectiveness study [7, 8]—are increasingly recognised as im- tions to complex social interventions, such as those in portant in the efficient evaluation of complex public public health. Progression criteria were highlighted as health interventions, our findings suggest that this area re- important by some as a means of preventing biased post mains in need of consistent standards to inform practice. hoc cases for continuation. However, there was a lack of At present, there are multiple definitions of exploratory guidance on how to devise progression criteria and pro- studies, a lack of consensus on a number of key issues, cesses for assessing whether these had been sufficiently and a paucity of detailed guidance on how to approach met. Where they had not been met, there was a lack of the main uncertainties such studies aim to address prior guidance on how to decide whether the exploratory to proceeding to a full evaluation. study had generated sufficient insight about uncertain- Existing guidance commonly focuses almost exclu- ties that the expense of a further feasibility study would sively on testing methodological parameters [33], such not be justified prior to large-scale evaluation. as recruitment and retention, although in practice, it is Although our review included a broad focus on guid- unusual for such studies not to also focus on the feasi- ance of exploratory studies from published and grey lit- bility of the intervention itself. Where intervention feasi- erature and moved beyond a focus on studies conducted bility is discussed, there is limited guidance on when an in preparation for an RCT specifically, a number of limi- intervention is ‘ready’ for an exploratory study and a lack tations should be noted. Guidance from other areas of of demarcation between intervention development and social intervention research where challenges may be pre-evaluation work to understand feasibility. Some similar to those in public health (e.g. education, social guidance recognised that an intervention continues to work and business) may not have been captured by our develop throughout an exploratory study, with distinc- search strategy. We found few worked examples of ex- tions made between ‘optimisation/refinement’ (i.e. minor ploratory studies in public health that provided substan- refinements to the intervention) vs. ‘major changes’. tial information from learned experience and practice. However, the point at which changes become so Hence, the review drew largely on recommendations Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 11 of 12 from funding organisations, or relatively abstract guid- Acknowledgements We thank the Specialist Unit for Review Evidence (SURE) at Cardiff University, ance from teams of researchers, with fewer clear exam- including Mala Mann, Helen Morgan, Alison Weightman and Lydia Searchfield, ples of how these recommendations are grounded in for their assistance with developing and conducting the literature search. experience from the conduct of such studies. As such, it Funding should be acknowledged that these documents represent This study is supported by funding from the Methodology Research Panel one element within a complex system of research pro- (MR/N015843/1). LM, SS and DW are supported by the UK Medical Research duction and may not necessarily fully reflect what is tak- Council (MC_UU_12017/14) and the Chief Scientist Office (SPHSU14). PC is supported by the UK Medical Research Council (MC_UU_12017/15) and the ing place in the conduct of exploratory studies. Finally, Chief Scientist Office (SPHSU15). The work was also undertaken with the treating sources of guidance as independent from each support of The Centre for the Development and Evaluation of Complex other does not reflect how some recommendations Interventions for Public Health Improvement (DECIPHer), a UKCRC Public Health Research Centre of Excellence. Joint funding (MR/KO232331/1) from developed over time (see for example [7, 8, 20, 36, 41]). the British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council, the Welsh Government and the Wellcome Trust, under the auspices of the UK Clinical Research Conclusion Collaboration, is gratefully acknowledged. There is inconsistent guidance, and for some key issues Availability of data and materials a lack of guidance, for exploratory studies of complex The datasets generated and/or analysed during the current study are not public health interventions. As this lack of guidance for publicly available due to copyright infringement. researchers in public health continues, the implications Authors’ contributions and consequences could be far reaching. It is unclear LM, GM, PC, MR, JS, RT and SS were involved in the development of the how researchers use existing guidance to shape decision- study. RT, JS, DW and BH were responsible for the data collection, overseen making in the conduct of exploratory studies, and in by LM and GM. Data analysis was undertaken by BH guided by RT, JS, DW and GM. The manuscript was prepared by BH, RT, DW, JS and GM. All doing so, how they adjudicate between various conflict- authors contributed to the final version of the manuscript. LM is the ing perspectives. This systematic review has aimed principal investigator with overall responsibility for the project. GM is Cardiff largely to identify areas of agreement and disagreement lead for the project. All authors read and approved the final manuscript. as a starting point in bringing order to this somewhat Ethics approval and consent to participate chaotic field of work. Following this systematic review, Not applicable. our next step is to conduct an audit of published public Competing interests health exploratory studies in peer-reviewed journals, to The authors declare that they have no competing interests. assess current practice and how this reflects the reviewed guidance. As part of a wider study, funded by Publisher’sNote the MRC/NIHR Methodology Research Programme to Springer Nature remains neutral with regard to jurisdictional claims in develop GUidance for Exploratory STudies of complex published maps and institutional affiliations. public health interventions (GUEST; Moore L, et al. Author details Exploratory studies to inform full scale evaluations of Centre for the Development and Evaluation of Complex Interventions for complex public health interventions: the need for guid- Public Health Improvement (DECIPHer), Cardiff University, Cardiff, Wales, UK. 2 3 Centre for Trials Research, Cardiff University, Cardiff, Wales, UK. MRC/CSO ance, submitted), the review has informed a Delphi sur- Social and Public Health Sciences Unit, University of Glasgow, Glasgow, UK. vey of researchers, funders and publishers of public Specialist Unit for Review Evidence, Cardiff University, Cardiff, Wales, UK. health research. In turn, this will contribute to a consen- Received: 6 February 2018 Accepted: 7 May 2018 sus meeting which aims to reach greater unanimity on the aims of exploratory studies, and how these can most efficiently address uncertainties which may undermine a References 1. Kessler R, Glasgow RE. A proposal to speed translation of healthcare full-scale evaluation. research into practice: dramatic change is needed. Am J Prev Med. 2011;40:637–44. 2. Sanson-Fisher RW, Bonevski B, Green LW, D’Este C. Limitations of the Additional files randomized controlled trial in evaluating population-based health interventions. Am J Prev Med. 2007;33:155–61. Additional file 1: Table S1. PRISMA checklist. (DOC 62 kb) 3. Speller V, Learmonth A, Harrison D. The search for evidence of effective Additional file 2: Appendix 1. Search strategies and websites. health promotion. BMJ. 1997;315(7104):361. Appendix 2. Coding framework. (DOCX 28 kb) 4. Arrowsmith J, Miller P. Trial watch: phase II failures: 2008–2010. Nat Rev Drug Discov. 2011;10(5):328–9. 5. National Institute for Health Research. Weight loss maintenance in adults Abbreviations (WILMA). https://www.journalslibrary.nihr.ac.uk/programmes/hta/084404/#/. CONSORT: Consolidated Standards of Reporting Trials; GUEST: GUidance for Accessed 13 Dec 2017. Exploratory STudies of complex public health interventions; MRC: Medical 6. Wight D, Wimbush E, Jepson R, Doi L. Six steps in quality intervention Research Council; NETSCC: National Institute of Health Research Evaluation development (6SQuID). J Epidemiol Community Health. 2015;70:520–5. Trials and Studies Coordinating Centre; NIHR: National Institute for Health 7. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. Research; PRISMA: Preferred Reporting Items for Systematic Reviews and CONSORT 2010 statement: extension to randomised pilot and feasibility Meta-Analyses; RCT: Randomised controlled trial trials. Pilot Feasibility Stud. 2016;2:64. Hallingberg et al. Pilot and Feasibility Studies (2018) 4:104 Page 12 of 12 8. Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman 29. Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex CL, et al. Defining feasibility and pilot studies in preparation for randomised intervention science: applying realist principles across all phases of the controlled trials: development of a conceptual framework. PLoS One. 2016; Medical Research Council framework for developing and evaluating 11:e0150205. complex interventions. Evaluation. 2016;22:286–303. 9. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL. Framework for design 30. Feeley N, Cossette S, Côté J, Héon M, Stremler R, Martorella G, et al. The and evaluation of complex interventions to improve health. BMJ. 2000; importance of piloting an RCT intervention. CJNR. 2009;41:84–99. 321(7262):694. 31. Levati S, Campbell P, Frost R, Dougall N, Wells M, Donaldson C, et al. Optimisation of complex health interventions prior to a randomised controlled 10. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. trial: a scoping review of strategies used. Pilot Feasibility Stud. 2016;2:1. Developing and evaluating complex interventions: new guidance. Medical 32. National Institute for Health Research. Feasibility and pilot studies. Available Research Council. 2008; from: http://www.nihr.ac.uk/CCF/RfPB/FAQs/Feasibility_and_pilot_studies. 11. National Institute for Health Research. The Filter FE Challenge: pilot trial and pdf. Accessed 14 Oct 2016. process evaluation of a multi-level smoking prevention intervention in 33. National Institute for Health Research. Glossary | Pilot studies 2015 http://www. further education settings. Available from: https://www.journalslibrary.nihr. nets.nihr.ac.uk/glossary?result_1655_result_page=P.Accessed 14 Oct 2016. ac.uk/programmes/phr/134202/#/. Accessed 25 Jan 2018. 34. Taylor RS, Ukoumunne OC, Warren FC. How to use feasibility and pilot trials 12. National Institute for Health Research. Adapting and piloting the ASSIST to test alternative methodologies and methodological procedures proir to model of informal peer-led intervention delivery to the Talk to Frank drug full-scale trials. In: Richards DA, Hallberg IR, editors. Complex interventions in prevention programme in UK secondary schools (ASSIST+Frank): an health: an overview of research methods. New York: Routledge; 2015. exploratory trial. https://www.journalslibrary.nihr.ac.uk/programmes/phr/ 35. Cook JA, Hislop J, Adewuyi TE, Harrild K, Altman DG, Ramsay CR et al. 12306003/#/. Accessed 25 Jan 2018. Assessing methods to specify the target difference for a randomised 13. Medical Research Council. A framework for the development and evaluation controlled trial: DELTA (Difference ELicitation in TriAls) review. Health of RCTs for complex interventions to improve health. London: Medical Technology Assessment (Winchester, England). 2014;18:v–vi. Research Council; 2000. 36. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: 14. Bonell CP, Hargreaves JR, Cousens SN, Ross DA, Hayes R, Petticrew M, et al. recommendations for good practice. J Eval Clin Pract. 2004;10:307–12. Alternatives to randomisation in the evaluation of public-health 37. National Institute for Health Research. Progression rules for internal pilot studies interventions: design challenges and solutions. J Epidemiol Community for HTA trials [14/10/2016]. Available from: http://www.nets.nihr.ac.uk/__data/ Health. 2009; https://doi.org/10.1136/jech.2008.082602. assets/pdf_file/0018/115623/Progression_rules_for_internal_pilot_studies.pdf. 15. O’Cathain A, Hoddinott P, Lewin S, Thomas KJ, Young B, Adamson J, et al. 38. Westlund E, Stuart EA. The nonuse, misuse, and proper use of pilot studies Maximising the impact of qualitative research in feasibility studies for in experimental evaluation research. Am J Eval. 2016;2:246–61. randomised controlled trials: guidance for researchers. Pilot Feasibility 39. Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. Stud. 2015;1(1):32. How we design feasibility studies. Am J Prev Med. 2009;36:452–7. 16. National Institute for Health Research. An exploratory trial to evaluate the 40. Strong LL, Israel BA, Schulz AJ, Reyes A, Rowe Z, Weir SS et al. Piloting effects of a physical activity intervention as a smoking cessation induction interventions within a community-based participatory research framework: and cessation aid among the ‘hard to reach’. https://www.journalslibrary. lessons learned from the healthy environments partnership. Prog nihr.ac.uk/programmes/hta/077802/#/. Accessed 13 Dec 2017. Community Health Partnersh. 2009;3:327–34. 17. National Institue for Health Research. Initiating change locally in bullying 41. Eldridge SM, Costelloe CE, Kahan BC, Lancaster GA, Kerry SM. How big and aggression through the school environment (INCLUSIVE): pilot should the pilot study for my cluster randomised trial be? Stat Methods randomised controlled trial. https://www.journalslibrary.nihr.ac.uk/hta/ Med Res. 2016;25:1039–56. hta19530/#/abstract. Accessed 13 Dec 2017. 42. Moffatt S, White M, Mackintosh J, Howel D. Using quantitative and 18. National Institute for Health Resarch. Increasing boys' and girls' intention to qualitative data in health services research—what happens when mixed avoid teenage pregnancy: a cluster randomised control feasibility trial of an method findings conflict? [ISRCTN61522618]. BMC Health Serv Res. 2006;6:1. interactive video drama based intervention in post-primary schools in 43. Hislop J, Adewuyi TE, Vale LD, Harrild K, Fraser C, Gurung T et al. Methods Northern Ireland. https://www.journalslibrary.nihr.ac.uk/phr/phr05010/ for specifying the target difference in a randomised controlled trial: the #/abstract. Accessed 13 Dec 2017. Difference ELicitation in TriAls (DELTA) systematic review. PLoS Med. 2014; 19. Arain M, Campbell, MJ, Cooper CL, Lancaster GA. What is a pilot or 11:e1001645. feasibility study? A review of current practice and editorial policy BMC Med Res Methodol. 2010;10:67. 20. Lancaster GA. Pilot and feasibility studies come of age! Pilot Feasibility Stud. 2015;1:1. 21. Shanyinde M, Pickering RM, Weatherall M. Questions asked and answered in pilot and feasibility randomized controlled trials. BMC Med Res Methodol. 2011;11:117. 22. Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;151:e1000097. 23. Dixon-Woods M, Agarwal S, Jones D. Intergrative approaches to qualitative and quantitative evidence. London: Health Development Agency; 2004. 24. Ritchie J, Spencer L, O’Connor W. Carrying out qualitative analysis. Qualitative research practice: a guide for social science students and researchers. 2003;1. 25. Möhler R, Bartoszek G, Köpke S, Meyer G. Proposed criteria for reporting the development and evaluation of complex interventions in healthcare (CReDECI): guideline development. IJNS. 2012;49(1):40–6. 26. Möhler R, Bartoszek G, Meyer G. Quality of reporting of complex healthcare interventions and applicability of the CReDECI list—a survey of publications indexed in PubMed. BMC Med Res Methodol. 2013;13:1. 27. Möhler R, Köpke S, Meyer G. Criteria for reporting the development and evaluation of complex interventions in healthcare: revised guideline (CReDECI 2). Trials. 2015;16(204):1. 28. Evans BA, Bedson E, Bell P, Hutchings H, Lowes L, Rea D, et al. Involving service users in trials: developing a standard operating procedure. Trials. 2013;14(1):1.

Journal

Pilot and Feasibility StudiesSpringer Journals

Published: May 28, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off