Background: One of the functions of the reformed Cancer Drugs Fund in England is as a managed access fund, providing conditional funding for cancer drugs where there is uncertainty in the economic case, and where that uncertainty can be addressed by data collection during two years’ use in the NHS. Our study characterises likely sources of such uncertainty, through a review of recent NICE Technology Appraisals. Methods: Discussions of uncertainty in NICE Appraisal Committees were extracted from published Single Technology Appraisals of cancer drugs, 2014–2016, and categorised inductively. The location of the comments within the structured Appraisal document was used as a proxy for the degree of concern shown by the Committee. Results: Twenty-nine appraisals were analysed, of which 23 (79%) were recommended for funding. Six main sources of uncertainty were identified. Immaturity of survival data, and issues relating to comparators, were common sources of uncertainty regardless of degree of concern. Uncertainties relating to quality of life, and the patient population in the trial, were discussed frequently but rarely occurred in the more uncertain appraisals. Concerns with trial design, and cost uncertainty, were less common, but a high proportion contributed to the most uncertain appraisals. Funding decisions were not driven by uncertainty in the evidence base, but by the expected cost per QALY relative to acceptance thresholds, and the resultant level of uncertainty in the decision. Conclusions: The reformed CDF is an improvement on its predecessor. However the main types of uncertainty seen in recent cancer appraisals will not readily be resolved solely by 2 years’ RWD collection in the reformed CDF; where there are no ongoing trials to provide longer-term data, randomised trials rather than RWD may be needed to fully resolve questions of relative efficacy. Other types of uncertainty, and concerns with generalisability, may be more amenable to the RWD approach, and it is these that we expect to be the focus of data collection arrangements in the reformed CDF. Keywords: Health technology appraisal, Cancer, Uncertainty, Cost-effectiveness, NICE, Cancer drugs fund * Correspondence: firstname.lastname@example.org Oxford-UCL Centre for the Advancement of Sustainable Medical Innovation, Radcliffe Department of Medicine, University of Oxford, Room 4403, Level 4, John Radcliffe Hospital, Headley Way, Headington, Oxford OX3 9DU, UK Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford, Old Road Campus, Roosevelt Drive, Headington, Oxford OX3 7LF, UK Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Morrell et al. BMC Health Services Research (2018) 18:393 Page 2 of 9 Background of our study was to identify likely sources of uncertainty, The Cancer Drugs Fund (CDF) in England was established through undertaking a review of NICE Technology in 2010, with the aim of improving access to new cancer Appraisals of cancer drugs in the years immediately pre- drugs, by providing funding for drugs not - or not yet - ceding the reform. recommended by the National Institute of Health and Care Excellence (NICE) for reasons of cost-effectiveness. Methods The Fund was intended as a temporary measure until a Our study is based on the expectation that the uncer- value-based pricing approach could be introduced. This tainties likely to be encountered by NICE in future can- entailed broadening the scope of NICE’s appraisal process cer drug evaluations, will be similar to those seen in to include additional elements of ‘value’, which would be submissions in the recent past, and will undergo similar taken into account in negotiating a drug’s price, and was review and debate in the NICE Appraisal Committees. expected to lead to more drugs being deemed sufficiently We therefore chose as our source, NICE’s Technology cost-effective for NHS use . However, the proposals Appraisal documents (Final Appraisal Document, FAD), failed to find broad stakeholder agreement, and were which publish the deliberations of the Committee and shelved; the CDF continued with expenditure rising from the evidence on which those discussions are based; that £200million at its inception in 2011/12, to over £400million is, our analysis reflects evaluation of uncertainty from in 2014/15 . the perspective of a decision-making committee. A time There has been extensive debate about the CDF’s justi- period of the 2 complete calendar years prior to the fication [3, 4] sustainability  and decision processes CDF reform proposals (March 2016) was chosen, to bal- , and in 2016 it underwent reform, under which all ance recency with sufficient number of cases. FADs for funding decisions were re-integrated into NICE. In cancer drugs for the period January 2014–March 2016 addition to providing interim funding for newly ap- were accessed via the NICE website. proved cancer drugs, the reformed CDF functions as a FADs follow a consistent template, within which the managed access fund. Specifically, where there is uncer- Committee’s deliberations are discussed in Section 4, tainty in the clinical and cost-effectiveness data such that ‘Consideration of the Evidence’, and this Section contains a drug cannot be recommended for routine commission- a summary table of the Committee’s key conclusions. ing, it can be recommended for funding through the Comments on uncertainty in these tables (both clinical CDF, providing: and cost-effectiveness uncertainty) were extracted, tabu- lated in Excel, and classified using an inductive process 1) 'The incremental cost-effectiveness ratios (ICERs) – that is, the categories were suggested by the text ra- presented have the plausible potential for satisfying ther than an imposed framework. A given piece of text the criteria for routine use […]; 2) It is possible that was classified unambiguously (linked to only one cat- the clinical uncertainty can be addressed through egory). We focus on the explicit meaning, and minimal collection of outcome data from patients treated in the inference is required in categorising; for example, discus- NHS; and 3) It is possible that the data collected sion of Kaplan-Meier curves is related unambiguously to (including from research already underway) will be survival analysis. Data extraction and classification was able to inform a subsequent update of the guidance. done by LM and reviewed by SW; both are health econ- This will normally happen within 24 months' . omists familiar with this technical vocabulary. We report the prevalence of each class of uncertainty as the num- The challenge of uncertainty in economic evaluation is ber of FADs in which it occurs; whilst there have been not new, and there are existing examples of conditional debates on the use of counts in content analysis, in this funding arrangements in the UK: for example, the Mul- case counts are valid as described by Hannah and tiple Sclerosis Risk-Sharing Scheme  and more re- Lausch  where the counted unit is clearly defined cently, managed entry schemes used for drugs for rare (the FAD), and differences in occurrence rates are conditions such as Duchenne muscular dystrophy . readily interpretable. Other TA information was also Elsewhere in Europe, examples include a coverage-with- extracted, including the funding decision, and the evidence-development route in the Netherlands  and Incremental Cost Effectiveness Ratio (ICER). a series of monitoring registries in Italy [10, 11]. From initial familiarisation with the FADs, we noted Given that the issue of uncertainty in economic evi- two particular locations where uncertainty was specific- dence is central to the new policy, it is important to ally discussed. Firstly, there are two rows in the sum- understand the uncertainties that might be expected to mary table which deal with uncertainty, in the clinical arise in candidates for CDF conditional funding, hence effectiveness and cost-effectiveness evidence respectively. the data required to reduce that uncertainty, and the ap- Secondly, the headline of the table (which reflects key propriate study designs for that data collection. The aim features of the appraisal overall) in some cases mentions Morrell et al. BMC Health Services Research (2018) 18:393 Page 3 of 9 uncertainty. To allow exploration of the strength of Submissions that had ICER estimates above the accept- concern with uncertainty, we assume that the uncertainty able range were Not Recommended, regardless of the rows of the table pick out the uncertainty issues of most level of uncertainty (for example pomalidomide in mul- concern during the Committee’s deliberations, and further tiple myeloma (MM) – all ICERs were above £50,000/ that the FADs with comments on uncertainty in the head- QALY ). line are those where uncertainty was a highly salient fea- We find no relationship between decision and either ture of the decision. The prevalence of the various sources the specific source of uncertainty, or the number of dif- of uncertainty was then considered over three levels, treat- ferent sources of uncertainty. For example, radium-223 ing location as a proxy for increasing concern: dichloride (prostate cancer) was recommended in a spe- cific subgroup despite multiple uncertainties (ICER ex- All appraisals, any comment on uncertainty in the pected to be within acceptable range) , and afatinib summary table (non-small cell lung cancer, NSCLC) was recommended All appraisals, comments from the specific by analogy to similar drugs despite no ICER being cal- uncertainty sections in the table culable . In contrast, trastuzumab emtansine (breast Only appraisals where uncertainty is specifically cancer) was rejected with low uncertainty across mul- discussed in the headline summary section. These tiple areas but a high ICER (£167,000/QALY) , whilst are referred to as ‘highly uncertain’ in our analysis. ramucirumab (gastric cancer) and pemetrexed (NSCLC) were rejected with significant uncertainty each in one Results particular area, also with high ICERs (£188,000– Thirty-three appraisals were published in the specified 408,000/QALY, £75,000/QALY respectively) [18, 19]. timeframe, of which four were terminated by NICE The main quantitative indicator of the uncertainty of the following non-submission of data by the manufacturer, decision, used in almost all the TAs, was the probability leaving 29 cases for analysis (21 solid tumours, eight of the technology being cost-effective at the relevant haematological malignancies). Of these, 18 were recom- cost-effectiveness threshold. mended for funding, five optimised (that is, recommended The most common types of uncertainty are described with restrictions relative to the licenced indication) and below, with illustrative examples from the reviewed the remaining six not recommended for funding. appraisals. We found that uncertainties in the evidence base were clearly signalled in the FADs by use of the word ‘uncer- Immature survival data tainty’, or terms such as: risk of bias, unreliable, weak, The uncertainty in these cases refers to a common immature, not generalisable. All 29 TAs discussed uncer- situation in oncology, where the trial data intended to tainty. The sources of uncertainty identified are shown establish the relative treatment effect of a new drug, in Table 1. Uncertainties in survival data and compara- extend over a period that is short relative to patients’ tors are common across all levels of uncertainty. Quality long-term survival; this ‘lifetime’ horizon is used in of Life (QoL) data and patient population are frequently cost-effectiveness modelling in order to capture all discussed, but are not as prevalent in the ‘highly uncer- the health effects for the QALY estimate . For ex- tain’ appraisals. Cost estimates and trial design are less ample, the aflibercept (metastatic colorectal cancer) commonly discussed, but around half of the instances submission was based on a median follow-up of 2 are found in the more uncertain appraisals. years, with uncertainty arising in the extrapolation Table 2 compares the appraisal decisions with level of out to 15 years . This issue is particularly relevant uncertainty as defined. The decisions do not appear to to overall survival; the cost-effectiveness models in reflect the level of uncertainty in the evidence base. the appraisals were typically some variant on a basic Rather, they reflect the estimated value of the ICER or three-state model: progression-free, progressed dis- ICER range, relative to the relevant cost-effectiveness ease, and death, hence key measures are progression- threshold. Drugs could be Recommended despite wide- free survival (PFS) and overall survival (OS). In the ranging ICER estimates, if those ICERs were expected to cases reviewed, generally a sufficiently large propor- fall within acceptable ranges; in such cases the uncer- tion of the cohort had progressed, such that there is tainty in the decision was low despite uncertainty in the a reasonable degree of certainty in the PFS. However, evidence base. For example: a smaller number will have died; for example, in the trial of bortezomib in mantle cell lymphoma, more ‘The Committee accepted that this ICER was than half of the patients in the trial were still alive at associated with uncertainty but, on balance, it was the time of analysis, so a median OS could not be satisfied that it would remain below £30,000 per calculated . There is therefore uncertainty in the QALY gained’ (enzalutamide in prostate cancer ) OS due to the need to extrapolate, with extensive Morrell et al. BMC Health Services Research (2018) 18:393 Page 4 of 9 Table 1 Analysis of comments on uncertainty in NICE appraisals of cancer drugs FAD: Final Appraisal Document AEs: adverse events FADs published January 2014–March 2016 debate on the appropriate statistical method chosen (described under section, Cost); if the optimum dur- for the extrapolation, as different models give rise to ation of treatment is not known it becomes challen- different projections for survival. Extrapolation of ging to estimate the associated survival, as in the overall survival can be further complicated by the appraisal of nivolumab in advanced melanoma . other uncertainties, such as duration of treatment Although PFS is a common surrogate outcome for OS, one example (bortezomib induction therapy in MM) also Table 2 Effect of the level of uncertainty on funding decisions directly used response rate; the aim of treatment is to allow Level of uncertainty more patients to proceed to stem cell transplant, and it was a b considered plausible that the observed effect of bortezomib Decision ‘High’ (n = 19) ‘Low’ (n = 10) on response rate could be associated with improved overall Recommended 11 7 survival . In the case of the erythropoiesis-stimulating Optimised 4 1 agents, extending survival is not the primary effect of treat- Not recommended 4 2 ment; the benefit is in avoiding the cost and disutility of High uncertainty: appraisals where uncertainty was discussed in the summary blood transfusion, which is sufficient to generate an accept- table headline Low uncertainty: all other evaluated appraisals able ICER . Morrell et al. BMC Health Services Research (2018) 18:393 Page 5 of 9 Lack of relevant comparator(s) there is reasonable expectation of racial or geographic In these examples, the drugs had not been tested directly differences. In addition a small number of the appraisals against the treatment(s) considered to be the relevant comment on the general concern that trial patients tend comparator(s) in England, as defined in the respective to be younger and fitter than the treated population. FAD. Examples included trials against alternatives no longer in routine use in England due to practice changes Quality of life data since the trials were done (9/16; for example, compari- Issues in this area are typically absence of health-related son of melanoma drugs against dacarbazine, which is Quality of Life (QoL) data – not collected in the trials, now rarely used ), or against placebo or best sup- collected using a non-comparable tool (eg ipilimumab in portive care where an active treatment is now used (2/16 melanoma: EORTC-CRC30 rather than EQ5D ), col- cases; for example comparison with best supportive care lected but not used in the submission (eg obinutuzumab in prostate cancer where docetaxel or abiraterone is now in CLL ). In these situations, values from the litera- used ). In three cases, practice across England was ture or alternative methods are used, and these have described as highly variable, resulting in a large number varying levels of validity. In one further case, health- of potential comparators which would have been infeas- related QoL was valued using a non-UK value set (pacli- ible to include in trials (eg pomalidomide in MM at taxel in pancreatic cancer: US value set for EQ5D ); third or subsequent relapse ). In the appraisals, the this creates debate but can readily be converted using relative effectiveness of the drugs is estimated using in- the raw data. direct treatment comparison or network meta-analysis. Cost Trial design The uncertainties in cost are varied in their reasons. The most common examples here were treatment diver- Uncertainties in survival on treatment, and optimal gences from licenced or current practice: use of doses duration of treatment, led to important uncertainties in (eg pembrolizumab in melanoma ) or regimens (eg drug cost (eg pembrolizumab in previously untreated nintedanib in NSCLC ) that differ from the licence, melanoma ). This uncertainty is not mitigated by or use of a different dose form (eg bortezomib in MM – Patient Access Schemes based on dose capping, because intravenous vs sub-cutaneous ). Use of crossover the average cost of the drug then becomes uncertain (eg from the control arm in the trial design was also import- lenalidomide in myelodysplastic syndrome: no drug cost ant, creating difficulties in attributing ultimate survival to NHS after 26 cycles ). Other cost uncertainties to the effect of a specific trial drug. A similar effect is include costs of treating adverse events, and the impact created by the effects of other drugs used later in the of vial sharing and dose reduction. treatment pathway, which was a source of uncertainty found infrequently in this study (Table 1). Other trial ef- Discussion fects include small or single arm studies (eg idelalisib in Common sources of uncertainty in technology appraisals untreated chronic lymphocytic leukaemia, CLL ), of cancer drugs during 2014–15 are the overall survival and debate on methods for establishing progression (eg estimates, and availability of relevant comparator data. olaparib in ovarian cancer ). Other sources of uncertainty are Quality of Life data, trial design, patient population, and costs. These findings are Population in the trials consistent with informal comments from current and These are examples where the patient sample in the tri- former committee members, and our observation of NICE als does not precisely match the licenced indication (eg Appraisal Committee meetings from the public gallery. pembrolizumab in melanoma: prior drug exposure ), Funding decisions do not appear to be driven by the level or the restricted indication under consideration in the or types of uncertainty per se, but by expected cost- TA; for example, a drug may be restricted to patients effectiveness relative to the cost-effectiveness threshold. with a specific mutation (eg erlotinib in NSCLC: EGFR- Neither of the two main types of uncertainty is likely TK mutation ), to a specific line of treatment (eg to be readily resolved by generating 2 years of ‘real olaparib in ovarian cancer: 4th or later line ) or a world’ data (RWD) within the CDF. There are difficulties particular tumour types (eg nintedanib: NSCLC with with use of observational data to generate evidence of adenocarcinoma histology ). The evaluation is there- relative effectiveness, as described by Grieve et al. . fore being performed using a post hoc secondary ana- In a randomised trial, randomisation allows outcomes to lysis of the trial data. There were further cases where the be compared between groups of patients who differ in trial locations did not include the UK (eg afatinib in treatment received, but are similar in other regards; in NSCLC: trials in an Asian population ) leading to contrast, in clinical use, patients are prescribed a given concerns with generalisability to a British population if therapy based on clinical characteristics, and are Morrell et al. BMC Health Services Research (2018) 18:393 Page 6 of 9 therefore systematically different from patients on any new CDF was osimertinib in NSCLC. The appraisal other treatment, thus introducing confounding into a identifies uncertainty in the overall survival extrapola- between-treatment comparison. Grieve et al. propose tion, and the generalisability of the trial data to UK clin- wider use of ‘only in research’ recommendations, to sup- ical practice. The data collection arrangements define port pragmatic randomised trials within the NHS . future analyses of the ongoing trials to resolve the sur- Our findings support this suggestion in situations where vival uncertainty, and focus the RWD collection on dur- the uncertainty concerns the relative treatment effect, ation of treatment and baseline characteristics of the and RWD carries the risk of selection bias. patient population . A more recent example (atezoli- The issue of confounding is highly relevant to uncer- zumab in urothelial carcinoma) similarly relies on an on- tainty relating to comparators, as the patients on the going Phase III trial for survival data, with NHS data comparator treatment in real-world use will be different collection on treatment duration . Of note, the prac- from those prescribed the drug of interest. Options for tice of linking the review of an appraisal to updates of establishing baseline survival on the current regimens the clinical data was evident before implementation of could include historical data from real-world use (such the reformed CDF; for example, nivolumab in melanoma as registry data prior to introduction of the new treat- has a scheduled review to coincide with updated survival ment), other trial data, or in-use data from other coun- data and studies on optimal treatment duration . tries; such sources need careful consideration for their Uncertainty in economic evaluation is typically de- generalisability to the current, UK, clinical population. scribed as four types [41, 42], summarised in Table 3.In For example, in the recent appraisal of avelumab in our analysis the main types of uncertainty observed were metastatic Merkel cell carcinoma, survival data for generalisability (patient populations), or related to the current standard of care was provided by an observa- assumptions and choices made in the estimates, which tional study; however no further comparator data can can be described as structural uncertainty (survival ex- now be generated as avelumab is changing that standard trapolation modelling, indirect and mixed treatment of care globally . comparisons). These are overlaid on inherent parameter For survival uncertainty, in addition to concerns with uncertainty, which can be characterised effectively using using RWD, we also need to consider the timeframe. In well established methods of probabilistic sensitivity ana- examples from this dataset, the submitted trials already lysis (PSA) and value of information analysis [42, 43]. In have 2+ years’ data, and yet there remains substantial contrast, structural uncertainty and generalisability have uncertainty in OS; 2 years’ de novo in-use data will pro- been less studied [44, 45] and are typically handled by vide no new information on long-term survival. In this one-way scenario analysis. Whilst this gives an indica- situation, the role of CDF funding is to allow clinical use tion of the impact of specific alternative assumptions, it whilst the original survival data matures in ongoing tri- cannot fully characterise the complex interactions of the als. This supports access to promising drugs showing a various sources of uncertainty without computing large marked improvement in survival, where the uncertainty paradoxically derives from the resulting low number of Table 3 Taxonomy of types of uncertainty in cost-effectiveness models events (progression or death) during the trials. However, there may be specific situations where 2 years’ RWD Type of Description Handled by: uncertainty could contribute to uncertainty reduction, such as where Parameter Uncertainty in estimates of Probabilistic sensitivity existing trials are small, or demonstrating a relationship uncertainty the values of the parameters analysis between survival and a surrogate marker. used in the cost-effectiveness Uncertainty due to drug regimens used in trials, small model, represented by the familiar concepts of standard sample size, patient population differences, QoL data, in- deviation and standard error cidence of AEs, and cost, may be more amenable to Structural The assumptions made in Sensitivity analysis resolution through RWD; for example, measuring dur- uncertainty constructing and populating ation of use or dosage patterns, where a comparison is cost-effectiveness models, not required, or where the data are to verify consistency such as the method used to extrapolate survival with predictions from another population or from a Methodological The analytical approaches Specification of a model. It is these types of uncertainty that we might ex- uncertainty used Reference Case of pect to see leading to conditional funding through the standard methods reformed CDF, and review of CDF entrants illustrates Generalisability To what extent the model, Sensitivity analysis that this has been the case; the majority involve survival assumptions and data data from ongoing trials, with RWD collection predom- represent the population for which the decision is being inantly on treatment patterns, and for generalisability. made For example, the first drug to be funded through the Morrell et al. BMC Health Services Research (2018) 18:393 Page 7 of 9 numbers of alternatives, and provides no indication of examples (for example PFS in advanced colorectal can- the likelihood of a given result, leading to high uncer- cer ). Hence validation of the relationship is essen- tainty of decision-making. Recent papers recommend tial, and that validation is specific to the tumour type, parameterising the uncertainty so it is reflected in the treatment, and treatment setting . Importantly cost-effectiveness model and hence in the PSA [44, 46, though, in the context of cost-effectiveness, the surro- 47], with Sculpher et al.  providing examples. Fur- gate outcome of PFS can have value in itself, potentially ther development of relevant methods or other decision adding QALY’s by extending time in a health state with support tools may be helpful for decision-makers faced high quality of life, and that time can be highly valued with this type of uncertainty. by patients . The reformed CDF goes part of the way towards the Beyond estimation of the probability that a technology proposals of Buxton et al. , in giving the CDF a spe- is cost-effective, we found little evidence of use of value- cific role in addressing issues of uncertainty, and in re- of-information analyses to support decisions. Such ana- integrating all cancer drug funding decisions under lysis is recommended in current frameworks for hand- NICE rather than providing an alternative funding ling uncertainty in decision-making [46, 47, 54]. These stream. Further, the fund’s pricing requirements provide frameworks are outlined in NICE’s Methods guide ( a mechanism for NICE to recommend drugs that might Section 6.4), so it is perhaps surprising not to see more otherwise have been rejected, thus providing health discussion of these concepts in the FADs. Sculpher et al. benefit for the population at a cost-effective price. These  discuss possible barriers, and suggest that the ap- features are improvements on the original CDF. The re- proaches could be used qualitatively in the absence of forms stop short, however, of proposing additional data formal analysis. It may be that NICEs committees are collection through new randomised trials, relying on considering these issues implicitly rather than using this RWD and ongoing trials, and this has been criticised as explicit terminology, so are not reported as such in the a missed opportunity to generate new, robust data for documents. Further, with the option of conditional reim- decision-making . bursement through the CDF, we may see more use of There is potential for improvement in approaches to such frameworks to guide decisions on the type and de- RWD. The UK in principle is well placed to generate rou- sign of further data collection. tine data; collection is centralised - described as the largest This study is limited by the relatively small number of of its type at its inception in 2013, with data extending appraisals included, and the secondary nature of the over 30 years - and reporting to the Systemic Anti-Cancer source, which has undergone condensation and inevitable Therapy dataset has been mandatory since 2014. This cap- filtering to produce the FAD. The FADs are produced by ability enables NHS data collection in the CDF managed NICE with the Committee chair and are reviewed by access agreements. There is however a need for clear Committee members; however there remains some risk of frameworks for integrating such data with trial evidence, inconsistent reporting between Committees. Using pri- and how the results will be used in decision-making when mary transcripts would avoid this risk, but require a compared to cost-effectiveness standards based on RCTs. higher level of interpretation by the researchers. Secondly, This could helpfully include direction on statistical most of the data extraction and classification was done by methods for accounting for selection bias. Further, it a single analyst. However, we were focusing on explicit would not be unreasonable to engage the broad range of content expressed in specific technical terms, rather than stakeholders – including patients – in design and use of requiring high levels of interpretation as in, for example, RWD, in the same way as for RCTs. The GetReal project thematic analysis of focus groups, where more than one – a European cross-stakeholder consortium – made simi- researcher would code and interpret themes. Our work lar observations in their recommendations for improving could be supplemented by formal interviews with commit- the use of real-world evidence . tee members and NICE staff. Finally, our study focused on Uncertainty in the evidence on overall survival is inev- the reported discussions of the NICE Appraisal Commit- itable, when trials are short relative to long-term sur- tees. Hence we do not address broader issues such as glo- vival. With current initiatives on earlier access (for bal clinical trial strategies, or the ability of current Quality example, the Accelerated Access Review and the UK’s of Life tools to capture the full range of patient experience; Early Access to Medicines Scheme), the challenges of these were not discussed in the appraisals we reviewed, dealing with uncertainty are likely to increase, with but clearly affect the availability and quality of data for growing reliance on surrogate outcomes . Although decision-making. surrogate outcomes have the advantage of providing re- sults more quickly, systematic reviews suggest that the Conclusion correlation between surrogates and OS in cancer is gen- The reformed CDF is an improvement on its predeces- erally low [50, 51], although stronger in some specific sor. However, the main types of uncertainty seen in Morrell et al. BMC Health Services Research (2018) 18:393 Page 8 of 9 recent cancer appraisals relate to overall survival esti- Author details Oxford-UCL Centre for the Advancement of Sustainable Medical Innovation, mates and availability of relevant comparator data. These Radcliffe Department of Medicine, University of Oxford, Room 4403, Level 4, will not readily be resolved solely by 2 years’ RWD col- John Radcliffe Hospital, Headley Way, Headington, Oxford OX3 9DU, UK. lection in the reformed CDF; where there are no on- Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford, Old Road Campus, Roosevelt Drive, Headington, going trials to provide longer-term data, randomised Oxford OX3 7LF, UK. Oxford NIHR Biomedical Research Centre, University of trials rather than RWD may be needed to fully resolve 4 Oxford, Oxford, UK. Department of Oncology, University of Oxford, Old Road questions of relative efficacy. Other types of uncertainty, Campus Research Building, Roosevelt Drive, Headington, Oxford OX3 7DQ, UK. Health Experiences Institute, Nuffield Department of Primary Care Health and concerns with generalisability, may be more amen- Sciences, University of Oxford, 23-38 Hythe Bridge Street, Oxford OX1 2ET, able to the RWD approach, and it is these that we expect UK. to be the focus of data collection arrangements in the re- Received: 9 August 2017 Accepted: 30 April 2018 formed CDF. We recommend further work on methods for characterisation of structural uncertainty, and con- tinued development of thinking on how observational References data can be best combined with other data types in cost- 1. The Parliamentary Office of Science and Technology. Value based effectiveness analysis. assessment of drugs. 2015. researchbriefings.files.parliament.uk/documents/ POST-PN-487/POST-PN-487.pdf. 2. National Audit Office. Investigation into the Cancer Drugs Fund. 2015. Abbreviations https://www.nao.org.uk/report/investigation-into-the-cancer-drugs-fund/. CDF: Cancer Drugs Fund; CLL: Chronic lymphocytic leukaemia; EoL: End of 3. Buxton M, Longworth L, Raftery J, Sculpher M, Towse A. Reforming the life; FAD: Final Appraisal Document; ICER: Incremental cost effectiveness ratio; Cancer drug fund. BMJ. 2014;349:g7276. https://doi.org/10.1136/bmj.g7276. MM: Multiple myeloma; NHS: National Health Service; NICE: National Institute 4. Linley WG, Hughes DA. Societal views on NICE, cancer drugs fund and for Health and Care Excellence; NSCLC: Non-small cell lung cancer; value-based pricing criteria for prioritising medicines: a cross-sectional OS: Overall survival; PAS: Patient access scheme; PFS: Progression free survey of 4118 adults in great Britain. Health Econ. 2013;22(8):948–64. survival; QALY: Quality-adjusted life year; QoL: Quality of life; RWD: Real-world 5. Dixon P, Chamberlain C, Hollingworth W. Did it matter that the Cancer data; TA: Technology Appraisal drugs fund was not NICE? A retrospective review. Value Health. 2016;19(6): 879–84. http://dx.doi.org/10.1016/j.jval.2016.04.001 Acknowledgements 6. NICE. PMG19 addendum a - final amendments to the NICE technology The authors thank Terry Hogan for automating data collection from the NICE appraisal processes and methods guides to support the proposed new website, and three reviewers for their helpful input. Cancer drugs fund arrangements. 2016 https://www.nice.org.uk/Media/ Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/ Funding process-and-methods-guide-addendum.pdf. This work was funded by a grant from the Policy Department, Cancer 7. McCabe C, Chilcott J, Claxton K, Tappenden P, Cooper C, Roberts J, et al. Research UK, to CASMI (LM, RWB). The funder had no role in the conduct of Continuing the multiple sclerosis risk sharing scheme is unjustified. BMJ. the study or writing of the manuscript, but has reviewed a draft. Open 2010;340 https://doi.org/10.1136/bmj.c1786. Access publishing is funded by the UK Charities Open Access Fund 8. NICE. Final evaluation determination: Ataluren for treating Duchenne allocation to the University of Oxford. muscular dystrophy with a nonsense mutation in the dystrophin gene. 2016 https://www.nice.org.uk/guidance/GID-DUCHENNEMUSCULARDYSTROPHY/ documents/final-evaluation-determination-document. Availability of data and materials 9. Boon W, Martins L, Koopmanschap M. Governance of conditional The datasets supporting the conclusions of this article are available from the reimbursement practices in the Netherlands. Health Policy. 2015;119(2):180– corresponding author on reasonable request. 5. http://dx.doi.org/10.1016/j.healthpol.2014.10.013 10. Montilla S, Xoxi E, Russo P, Cicchetti A, Pani L. Monitoring registries at Italian Authors’ contributions Medicines Agency : fostering access, guaranteeing sustainability. Int J Technol LM designed the study, generated and analysed the data, and drafted the Assess Health Care. 2015;31(4):210–3. https://doi.org/10.1017/S0266462315000446. manuscript. SW and SR contributed to the study design and reviewed data 11. Faulkner SD, Lee M, Qin D, Morrell L, Xoxi E, Sammarco A, et al. Pricing extraction and classification. AS and MRM provided interpretation and clinical and reimbursement experiences and insights in the European Union perspective. SW, SR, AS, MRM and RB provided critical review of the and the United States: lessons learned to approach adaptive payer manuscript. All authors read and approved the final manuscript. LM acts as pathways. Clinical Pharmacology & Therapeutics. 2016;100(6):730–42. overall guarantor. https://doi.org/10.1002/cpt.508. 12. Hannah D, Lautsch B. Counting in qualitative research: why to conduct it, Ethics approval and consent to participate when to avoid it, and when to closet it. Journal of Management Enquiry. Not applicable – this article does not contain any studies with human 2011;20(1):14–22. https://doi.org/10.1177/1056492610375988. participants or animals performed by any of the authors. 13. NICE. Enzalutamide for metastatic hormone-relapsed prostate cancer previously treated with a docetaxel-containing regimen (TA316) 2014. Competing interests https://www.nice.org.uk/guidance/ta316. AS has received consulting fees or honoraria from Roche, Gilead, Abbvie, 14. NICE. Pomalidomide for relapsed and refractory multiple myeloma Janssen and Novartis. MRM has received consulting fees from previously treated with lenalidomide and bortezomib (TA338). 2015. https:// pharmaceutical companies with CDF-funded drugs, including Roche, Merck, www.nice.org.uk/guidance/ta338. Novartis, BMS and GSK, and has conducted clinical trials for which his institu- 15. NICE. Radium-223 dichloride for treating hormone-relapsed prostate tion is funded by the pharmaceutical industry. SR has received fees from cancer with bone metastases (TA412) 2016. https://www.nice.org.uk/ Takeda for leadership training. RWB is a non-executive director of Celgene guidance/ta412. Corporation. LM and SW declare that they have no conflict of interest. 16. NICE. Afatinib for treating epidermal growth factor receptor mutation- positive locally advanced or metastatic non-small-cell lung cancer (TA310) 2014. https://www.nice.org.uk/guidance/ta310. Publisher’sNote 17. NICE. Trastuzumab emtansine for treating HER2-positive, unresectable locally Springer Nature remains neutral with regard to jurisdictional claims in advanced or metastatic breast cancer after treatment with trastuzumab and a published maps and institutional affiliations. taxane (TA371) 2015. https://www.nice.org.uk/guidance/ta371. Morrell et al. BMC Health Services Research (2018) 18:393 Page 9 of 9 18. NICE. Ramucirumab for treating advanced gastric cancer or gastro– 44. Bojke L, Claxton K, Sculpher M, Palmer S. Characterizing structural oesophageal junction adenocarcinoma previously treated with uncertainty in decision analytic models: a review and application of chemotherapy (TA378) 2016. https://www.nice.org.uk/guidance/ta378. methods. Value Health. 2009;12(5):739–49. https://doi.org/10.1111/j.1524- 19. NICE. Pemetrexed maintenance treatment following induction therapy with 4733.2008.00502.x. pemetrexed and cisplatin for non-squamous non-small-cell lung cancer 45. Sculpher MJ, Pang FS, Manca A, Drummond MF, Golder S, Urdahl H, et al. (TA309) 2014. https://www.nice.org.uk/guidance/ta309. Generalisability in economic evaluation studies in healthcare: a review and 20. NICE. Guide to the methods of technology appraisal 2013. https://www. case studies. Health technology assessment (Winchester, England). 2004; nice.org.uk/process/pmg9/chapter/foreword. 8(49):iii–v. 1-192 46. Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. 21. NICE. Aflibercept in combination with irinotecan and fluorouracil-based Model parameter estimation and uncertainty analysis: a report of the ISPOR- therapy for treating metastatic colorectal cancer that has progressed SMDM modeling good research practices task force working group–6. Med following prior oxaliplatin-based chemotherapy (TA307) 2014. https://www. Decis Mak. 2012;32(5):722–32. https://doi.org/10.1177/0272989X12458348. nice.org.uk/guidance/ta307. 47. Sculpher M, Basu A, Kuntz K, Meltzer D. Reflecting uncertainty in cost- 22. NICE. Bortezomib for previously untreated mantle cell lymphoma (TA370). effectiveness analysis. In: Neumann PJSG, Russell LB, Siegel JE, Ganiats TG, 2015 https://www.nice.org.uk/guidance/ta370. editors. Cost-effectiveness in health and medicine (2nd edition). New York: 23. NICE. Nivolumab for treating advanced (unresectable or metastatic) Oxford University Press; 2017. melanoma (TA384) 2016. https://www.nice.org.uk/guidance/ta384. 48. GetReal I. Advancing evidence generation for new drugs: IMI GetReal’s 24. NICE. Bortezomib for induction therapy in multiple myeloma before high- Recommendationson real-world evidence2017. http://www.imi-getreal.eu/ dose chemotherapy and autologous stem cell transplantation (TA311) 2014. Publications/Deliverables-and-reports. https://www.nice.org.uk/guidance/ta311 . 49. Kemp R, Prasad V. Surrogate endpoints in oncology: when are they acceptable 25. NICE. Erythropoiesis-stimulating agents (epoetin and darbepoetin) for for regulatory and clinical decisions, and are they currently overused? BMC treating anaemia in people with cancer having chemotherapy (TA323) 2014. Med. 2017;15:134. https://doi.org/10.1186/s12916-017-0902-9. https://www.nice.org.uk/guidance/ta323. 50. Ciani O, Davis S, Tappenden P, Garside R, Stein K, Cantrell A, et al. Validation 26. NICE. Dabrafenib for treating unresectable or metastatic BRAF V600 mutation- of surrogate endpoints in advanced solid tumors: systematic review of positive melanoma (TA321) 2014. https://www.nice.org.uk/guidance/ta321. statistical methods, results, and implications for policy makers. Int J Technol 27. NICE. Pembrolizumab for advanced melanoma not previously treated with Assess Health Care. 2014;30(3):312–24. https://doi.org/10.1017/ ipilimumab (TA366) 2015. https://www.nice.org.uk/guidance/ta366. s0266462314000300. 28. NICE. Nintedanib for previously treated locally advanced, metastatic, or 51. Prasad V, Kim C, Burotto M, Vandross A. The strength of association locally recurrent non-small-cell lung cancer (TA347) 2015. https://www.nice. between surrogate end points and survival in oncology: a systematic review org.uk/guidance/ta347. of trial-level meta-analyses. JAMA Intern Med. 2015;175(8):1389–98. https:// 29. NICE. Idelalisib for treating chronic lymphocytic leukaemia (TA359)2015. doi.org/10.1001/jamainternmed.2015.2829. https://www.nice.org.uk/guidance/ta359. 52. Ciani O, Buyse M, Drummond M, Rasi G, Saad ED, Taylor RS. Time to review 30. NICE. Olaparib for maintenance treatment of relapsed, platinum-sensitive, the role of surrogate end points in health policy: state of the art and the BRCA mutation-positive ovarian, fallopian tube and peritoneal cancer after way forward. Value Health. 2017;20(3):487–95. https://doi.org/10.1016/j.jval. response to second-line or subsequent platinum-based chemotherapy 2016.10.011. (TA381)2016. https://www.nice.org.uk/guidance/ta381. 53. NICE. Palbociclib with an aromatase inhibitor for previously untreated, 31. NICE. Pembrolizumab for treating advanced melanoma after disease hormone receptor-positive, HER2-negative, locally advanced or metastatic progression with ipilimumab (TA357)2015. https://www.nice.org.uk/ breast cancer (TA495) 2017. https://www.nice.org.uk/guidance/ta495. guidance/ta357. 54. Claxton K, Palmer S, Longworth L, Bojke L, Griffin S. Informing a decision 32. NICE. Erlotinib and gefitinib for treating non-small-cell lung cancer that has framework for when NICE should recommend the use of health progressed after prior chemotherapy (TA374)2015. https://www.nice.org.uk/ technologies only in the context of an appropriately designed programme guidance/ta374. of evidence development. Health Technol Assess. 2012;16(46):323. https:// 33. NICE. Ipilimumab for previously untreated advanced (unresectable or doi.org/10.3310/hta16460. metastatic) melanoma (TA319) 2014. https://www.nice.org.uk/guidance/ta319. 34. NICE. Obinutuzumab in combination with chlorambucil for untreated chronic lymphocytic leukaemia (TA343) 2015. https://www.nice.org.uk/guidance/ta343. 35. NICE. Paclitaxel as albumin-bound nanoparticles in combination with gemcitabine for previously untreated metastatic pancreatic cancer (TA360) 2015. https://www.nice.org.uk/guidance/ta360. 36. NICE. Lenalidomide for treating myelodysplastic syndromes associated with an isolated deletion 5q cytogenetic abnormality (TA322) 2014. https:// www.nice.org.uk/guidance/ta322. 37. Grieve R, Abrams K, Claxton K, Goldacre B, James N, Nicholl J, et al. Cancer drugs fund requires further reform. BMJ. 2016;354:i5090. 38. NICE. Avelumab for merkel cell carcinoma (ID1102) 2018. https://www.nice. org.uk/guidance/indevelopment/gid-ta10136/documents. 39. NICE. Osimertinib for treating locally advanced or metastatic EGFR T790M mutation-positive non-small-cell lung cancer: managed access agreement. 2016. https://www.nice.org.uk/guidance/ta416/resources . 40. NICE. Atezolizumab for untreated locally advanced or metastatic urothelial cancer when cisplatin is unsuitable (TA492) 2017. https://www.nice.org.uk/ guidance/ta492/resources.. 41. Briggs A. Handling uncertainty in economic evaluation and presenting the results. In: Drummond MF, McGuire A, editors. Economic evaluation in health care: merging theory and practice. Oxford: Oxford University Press; 2001. p. 172–214. 42. Briggs A, Claxton K, Sculpher M. Decision modelling for health economic evaluation. Handbooks in health economic evaluation. Oxford: Oxford University Press; 2006. 43. Gray A, Clarke P, Wolstenholme J, Wordsworth S. Applied methods of cost- effectiveness analysis in health care. Handbooks in HealthEconomic evaluation. Oxford: Oxford University Press; 2011.
BMC Health Services Research – Springer Journals
Published: May 31, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera