Applying a Multicriteria Decision Analysis (MCDA) Approach to Elicit Stakeholders’ Preferences in Italy: The Case of Obinutuzumab for Rituximab-Refractory Indolent Non-Hodgkin Lymphoma (iNHL)

Applying a Multicriteria Decision Analysis (MCDA) Approach to Elicit Stakeholders’ Preferences... PharmacoEconomics Open (2018) 2:153–163 https://doi.org/10.1007/s41669-017-0048-x OR IGINAL RESEARCH ARTIC L E Applying a Multicriteria Decision Analysis (MCDA) Approach to Elicit Stakeholders’ Preferences in Italy: The Case of Obinutuzumab for Rituximab-Refractory Indolent Non-Hodgkin Lymphoma (iNHL) 1 1 1 2 • • • • Martina Garau Grace Hampson Nancy Devlin Nicola Amedeo Mazzanti Antonio Profico Published online: 22 August 2017 The Author(s) 2017. This article is an open access publication Abstract Conclusions MCDA can be used to elicit the views of Background Healthcare decision makers need to make different stakeholder groups and has the potential to trade-offs between different elements of value of new structure and inform reimbursement decisions. treatments. Multicriteria decision analysis (MCDA) pro- vides a framework that can help decision makers to understand stakeholders’ preferences and be explicit about Key Points for Decision Makers the trade-offs that are being made. Objective The objective of this study was to use MCDA to Multicriteria decision analysis (MCDA) provides a obtain preferences and views on decision criteria across framework that can help decision makers to three stakeholder groups (patients, clinicians and payers) in understand stakeholders’ preferences and be explicit Italy and to use these to assess the performance of obinu- about the trade-offs that are being made between tuzumab for rituximab-refractory indolent non-Hodgkin different elements of value. lymphoma (iNHL). Based on a convenience sample, we found that Methods We used EVIDEM V3.0, an MCDA framework, patients and clinicians give greater weight to and collected participants’ preferences via an online survey targeting severe conditions; payers are most and structured meetings. concerned with unmet need, comparative costs, and Results Patients and clinicians expressed a preference for high-quality evidence. interventions targeting severe conditions. Payers expressed preference for treatments targeting areas with an unmet The overall value score of obinutuzumab based on need, which are cheaper than the comparator, and with all stakeholder groups’ responses combined was high-quality evidence. Obinutuzumab in combination with mainly driven by the criteria disease severity, type of bendamustine, compared with bendamustine alone, therapeutic benefit and unmet needs. received high positive scores for the criteria ‘disease severity’ and ‘type of therapeutic benefit’ by all three groups, and negative scores on the economic-related cri- teria, according to all stakeholder groups. 1 Background Electronic supplementary material The online version of this article (doi:10.1007/s41669-017-0048-x) contains supplementary material, which is available to authorized users. Stakeholders such as patients can be involved in medicine reimbursement decisions and Health Technology Assess- & Martina Garau ment (HTA) in different ways, from contributions to evi- mgarau@ohe.org dence submissions to participation in advisory committees. Office of Health Economics, London, UK In some HTA processes, such as that of the National Institute for Health and Care Excellence (NICE) in Roche S.P.A, Milan, Italy 154 M. Garau et al. England and Wales, and the Canadian Agency for Drugs 2 Methods and Technologies in Health (CADTH) in Canada, patients submit evidence and participate in committee meetings [1]. There are number of approaches to eliciting preferences The Scottish Medicines Consortium (SMC) in Scotland has with different levels of complexity and theoretical bases, introduced the option of consulting a panel of clinicians including swing weights, analytic hierarchy process and and patient representatives during the assessment of discrete choice experiments [6]. Recently, there has also medicines for rare or end-of-life conditions [2]. been a proliferation of frameworks to assess the value of HTA decision makers need to understand how stake- new interventions, developed by organisations including holders trade-off between different types of benefit poten- the American Society of Clinical Oncology (ASCO), the tially generated by a new treatment. Stakeholders need to European Society for Medical Oncology (ESMO) and the understand to what extent and how their input affects final Institute for Clinical and Economic Review (ICER) [10]. HTA decisions. A study of appraisals in five countries [3] For the purposes of this study, we selected the EVIDEM found that patient preferences are rarely mentioned in HTA approach, which combines a coherent set of criteria with reports, and that patient participation in HTA tend to be tools to operationalise it in decision making (including unsystematic. suggested approaches for preference elicitation). In addi- Multicriteria decision analysis (MCDA) provides a clear tion, EVIDEM was specifically designed for healthcare framework to assess the value of a treatment compared decision making. Its choice here enables results to be with alternative treatments or standard of care (SoC), compared with a growing literature from its use in other against multiple and competing criteria [4]. It can support healthcare decision-making settings. decision makers to be explicit about the trade-offs made EVIDEM (V3.0) is an open-source framework resulting between the selected criteria, and offers systematic and from a collaboration of experts and stakeholders, and is robust ways to elicit preferences and consider evidence subject to continued testing and adaptations [11]. In the last from stakeholders. For example, in Germany, the Institute 10 years, the EVIDEM collaboration has merged the for Quality and Efficiency in Health Care (IQWiG) ran practical aspects of developing an implementable frame- MCDA pilots to elicit patient preferences on clinical out- work with ethical foundations for its criteria and objectives. comes in depression and hepatitis C [5, 6]. In Italy, the In particular, ethical justification has been provided to the Lombardia region uses an MCDA framework to select criteria, including ‘‘distributive justice and fairness (pri- health technologies to reimburse [7]. oritise those who are worst off)’’ [11]. EVIDEM also The purpose of this study was to use an MCDA advocates for important procedural values such as ‘partic- approach to obtain preferences and views on criteria and on ipatory decision making’, which aligns with the ethical performance of obinutuzumab for rituximab-refractory framework of accountability for reasonableness developed indolent non-Hodgkin lymphoma (iNHL) across three by Daniels and Sabin for ‘legitimate’ priority settings stakeholder groups (patients, clinicians and payers) in Italy. [11, 12]. As far we know, this is the first study exploring stake- EVIDEM comprises a broad range of criteria, capturing holders’ preferences to inform national reimbursement elements of value relevant to patients, healthcare systems decisions of medicines in Italy. While Castro et al. describe and society, with properties (including non-overlap the use of EVIDEM in Italy in the context of a regional between criteria), and with operational purposes, enabling payer and focused on medical devices, we conducted our us to inform real-life decision making. EVIDEM has been MCDA exercise to inform value assessment and decision applied in HTA in a number of jurisdictions, including making at the national level, specifically in the context of Canada [13] and, more recently, Spain, where the HTA medicines [8], which is currently under the responsibility body in Catalonia has explored the framework for of Agenzia Italiana del Farmaco (AIFA ). appraising orphan drugs [14]. The Italian region Lombardia Our work built on the study by Wahlster et al. [9], who uses EVIDEM to inform local funding decisions of health applied EVIDEM to capture stakeholders’ preferences in interventions. Our aim was to extend the application of Germany using an intervention case study. In addition to MCDA to support national decision making of medicine the online survey that was used to obtain preferences reimbursement. individually by Wahlster et al., we applied EVIDEM to EVIDEM includes a set of 13 clearly defined and support group discussions. This is a fundamental role that measurable ‘core’ criteria grouped into domains. Some MCDA can play in HTA to structure committees’ consid- criteria are measured in absolute terms, not relative to other eration of different and often conflicting perspectives. interventions, and other criteria are measured compara- tively to existing interventions. 1 2 http://www.agenziafarmaco.gov.it/en. https://www.evidem.org/. Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 155 Figure 1 presents the core EVIDEM framework, because it combines simplicity with the ability of ‘forcing’ including five domains: need for intervention, comparative people to prioritise criteria. We also noted that van Til et al. outcomes of intervention, type of benefit, economic con- [15] showed that the choice of the weight elicitation sequences of intervention, and knowledge about interven- method does not affect value estimates at the group level. tion. Each domain comprises a set of criteria. The description of the criteria and the survey instructions In May 2016, an online survey (using SurveyMonkey ) given to participants are available in the electronic sup- was sent to three groups of stakeholders involved or plementary material. To obtain one set of weights for each affected by reimbursement decisions in Italy: clinicians, criterion, we combined domain weights with those within patients, and payers. This step of the process was labelled each domain and normalised the values (to sum up to 1). as the ‘survey first round’. The survey was used to elicit Respondents were asked to score the performance of preferences around the relative importance of criteria obinutuzumab in combination with bendamustine followed (weights), and the degree of achievement of obinutuzumab by obinutuzumab maintenance, compared with ben- against these criteria compared with the current SoC damustine alone (which is the only efficacious intervention (scores). in this indication [16]), in patients with rituximab-refrac- We derived weights using the ‘point allocation’ tory iNHL. This is in line with obinutuzumab license approach, where we asked participants to allocate 100 indication and clinical evidence [16, 17]. points, first across criterion domains and, second, across Incremental criteria, related to the health and non-health criteria within each clusters. This method was selected effects of the intervention, were measured on a scale from Fig. 1 Structure of the MCDA EVIDEM framework, including all criteria and categories 156 M. Garau et al. -5to ?5. Absolute criteria were measured on a scale from Participants were drawn from existing manufacturer 0to5. networks. The payer group involved hospital, regional and Evidence on obinutuzumab for iNHL was based on lit- national decision makers; patient representatives were erature reviews included in HTA submissions and results of members of Italian patient groups related to lymphomas; obinutuzumab clinical trials. Evidence was reviewed and and clinicians were lymphoma specialists. Members of synthesised using the EVIDEM framework. The evidence each group were distributed across Italian regions, covering matrix provided to participants is presented in the elec- the north, centre, and south areas. tronic supplementary material. Most of the evidence for the ‘comparative efficacy/ef- fectiveness’ criterion referred to one of the clinical end- 3 Results points measured in the clinical trial, i.e. progression-free survival [16]. We recognise that this is a surrogate endpoint A total of 19 people were invited and completed the first with different levels of associations with overall survival round of the survey, including nine patients, five clinicians (OS), which is the primary measure of efficacy [18]. and five payers. The structured meetings were found to be Nevertheless, there is an increasing acceptance of surrogate useful by participants to discuss the MCDA framework and endpoints by regulatory agency and other healthcare deci- study approach with their peers. Only the clinician group sion makers, given the additional time and resources reviewed their answers following the structured meetings. required for collecting evidence to estimate OS. Responses were analysed in Microsoft Excel (Microsoft 3.1 Stakeholders’ Weights Corporation, Redmond, WA, USA), and key results, including average and standard deviation (SD) of weights Weights represent trade-offs between criteria [6] and thus and score values from the survey first round, were pre- reveal which aspects of value matter most to each group. sented at structured meetings. Structured meetings, one To obtain ‘generic’ weights to apply to any intervention, with each stakeholder group, were run to allow participants respondents were required to express their preferences to seek further explanation on the MCDA framework, to between the EVIDEM criteria, based solely on their defi- discuss and review weights and scores obtained, identify nition, and scale measurement, not the description of the areas of agreement and disagreement among participants, intervention. and, where possible, reach a consensus on the weight and Figure 2 compares the sets of normalised weights from score values that could best represent the group’s each group. According to patients, the two most important perspective. criteria were the ‘type of therapeutic benefit’ and ‘disease To minimise the participants’ cognitive burden and have severity’, both with weights of 11% (SD 0.07 and 0.10, a manageable number of criteria, the meeting discussions respectively). These are both absolute criteria (not relative concentrated on the EVIDEM core criteria (presented in to comparative interventions). The three least important Fig. 1) and did not include the contextual qualitative cri- criteria were the three economic indicators: ‘comparative teria, which were omitted from the final results. The con- non-medical costs’, ‘comparative other medical costs’, and textual criteria do not require weights and scores, therefore ‘comparative cost of intervention’, with weights of 3% (SD their exclusion do not affect the overall value score. 0.03), 4% (SD 0.04) and 5% (SD 0.04), respectively. Participants were invited to complete the survey in light According to clinicians, the two most important criteria of the discussion at the meeting (survey second round). were ‘disease severity’ and ‘comparative efficacy/effec- Following this, the average of the normalised weights and tiveness’, with weights of 15% (SD 0.10) and 12% (SD scores from the three groups were combined with linear 0.05), respectively. The three least important criteria were aggregation to calculate the intervention value score. The two economic indicators (‘comparative non-medical costs’ literature suggests a variety of approaches to aggregating and ‘comparative other medical costs’), and ‘type of pre- the preferences and views expressed by individuals to ventative benefit’, with weights of 3% (SD 0.02), 4% (SD inform group decision making. These include agreeing the 0.02) and 4% (SD 0.03), respectively. weight and score values as part of the committee discus- Finally, payers indicated that the three most important sion; aggregating by using the average of weights of scores criteria were ‘unmet needs’, ‘comparative cost of inter- obtained from responders; and retaining and comparing vention’, and ‘quality of evidence’, with weights of 11% respondents’ values [19]. We implemented the third (SD 0.07), 11% (SD 0.05) and 10% (SD 0.02), respec- approach as we observed the differences between groups’ tively. The three least important criteria were ‘comparative and individuals’ values. We also used the second approach non-medical costs’, ‘size of affected population’ and as an example of incorporating stakeholders’ preferences in ‘comparative other medical costs’, with scores of 4% (SD value assessment. 0.01), 5% (SD 0.02) and 5% (SD 0.02), respectively. The Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 157 Fig. 2 Relative weights of Type of therapeuc benefit individual criteria by Disease severity stakeholder group Comparave paent-received health/PRO Comparave effecveness Type of prevenve benefit Expert consensus/clinical pracce guidelines Comparave safety/tolerability Unmet needs Size of affected populaon Quality of evidence Comparave cost consequences - cost of intervenon Comparave cost consequences - other medical costs Comparave cost consequences - non-medical costs 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 Mean normalized weights Clinician Payer Paent small SDs reflect a high level of agreement within this quality evidence. In contrast, the quality of evidence cri- stakeholder group about the importance of each criteria, terion was deemed a low priority by patients. despite a small sample size. Given the limited sample size, we did not perform 3.2 Stakeholders’ Scores any statistical comparison across stakeholder group weights; however, it is worth highlighting key differ- Based on different outcome measures and types of evi- ences and commonalities. Compared with patients and dence on obinutuzumab for the treatment of rituximab-re- clinicians, payers distributed the weights more equally fractory iNHL, participants provided scores. Unlike among the domains. Payers’ weights range between 17 weights, scores are specific to the intervention under con- and 24%, compared with those given by clinicians and sideration. Figure 3 presents the scores allocated by the patients, which range between 12 and 33%. These two three stakeholder groups. groups give less weight to the domains ‘economic con- Patients assigned the highest scores (representing the sequences of intervention’ and ‘knowledge about the areas where obinutuzumab performs best) to ‘unmet intervention’. needs’, with an average score (AS) of 3.7 (SD 0.65); Patients’ and clinicians’ views were aligned as they ‘disease severity’, with an AS of 3.6 (SD 0.46); and ‘type expressed preference for interventions targeting severe of therapeutic effect’, with an AS of 3.1 (SD 1.05). The conditions. The highest weights in both group were criteria part of the ‘comparative outcomes of intervention’ observed for this criterion. Patients also believed that pri- domain were, on average, all positive, indicating that ority should be given to interventions that have a signifi- obinutuzumab is expected to generate incremental health cant therapeutic effect (for example, they offer a cure or gains compared with SoC. However, in the criterion ‘pa- significantly delay progression of the disease), while clin- tient-perceived health/patient-reported outcomes’, there icians indicated that one of the most important criteria is was a large variation in the assigned scores, which ranged the improvement in clinical outcomes compared with SoC. from -2to ?5. All three criteria related to the economic Both groups ranked all the economic criteria among the impact of implementing the intervention were scored five least important criteria. negative, on average. Payers allocated higher weights to the economic-related Areas in which obinutuzumab was deemed to perform criteria, with the direct (incremental) cost of the interven- better than the comparator by clinicians were ‘size of tion being one of the most important. Their preferences affected population’, with an AS of 4 (SD 0.71); and were for treatments targeting populations in which there is ‘disease severity’, ‘type of therapeutic effect’, and ‘quality little or no effective treatment, which are less expensive of evidence’, with an AS of 3.6 (SD 0.89, 0.89, 0.55, than the comparator, and which are underpinned by high- respectively). Two criteria part of the ‘comparative 158 M. Garau et al. Fig. 3 Mean scores for Scores for obinutuzumab for the three stakeholder groups obinutuzumab for each criteria Type of therapeuc benefit by stakeholder group Disease severity Comparave paent-received health/PRO Comparave effecveness Type of prevenve benefit Expert consensus/clinical pracce guidelines Comparave safety/tolerability Unmet needs Size of affected populaon Quality of evidence Comparave cost consequences - cost of intervenon Comparave cost consequences - other medical costs Comparave cost consequences - non-medical costs -4 -3 -2 -1 0123 4 Clinician Payer Paent outcomes of intervention’ domain were, on average, posi- whose patent has recently expired, according to all tive. However, in terms of safety and tolerability, obinu- stakeholder groups. This is because the cost of obinu- tuzumab was deemed slightly worse than its comparator tuzumab in combination with bendamustine and related [with a score of -0.4 (SD 0.89)]. Clinicians also assigned medical costs was higher than those of bendamustine negative scores to obinutuzumab against ‘comparative cost alone. of intervention’ and ‘comparative other medical costs’, which obtained a score of -1.2 (SD 1.10) and -0.4 (SD 3.3 Overall Value Score of Obinutuzumab 0.89), respectively. Payers gave the highest scores to ‘disease severity’ and To develop a combined perspective on obinutuzumab ‘unmet needs’, with an AS of 4.0 (SD 0.71, 0.71, respec- value, all survey responses were included and weighted tively); and ‘type of therapeutic benefit’, with an AS of 3.4 equally. As shown in Fig. 4, the value score of obinu- (SD 0.55). Two criteria received negative scores, on tuzumab was 0.45. average: ‘comparative cost of intervention’ and ‘compar- A number of MCDA best-practice articles challenge the ative other medical costs’. inclusion of (incremental) costs as a separate criterion. If We observed some consistency across stakeholder the overall score is a composite measure of benefit, costs groups in relation to obinutuzumab scores. are not an attribute of benefit [20]. In addition, this would The criteria ‘disease severity’ and ‘type of therapeutic not allow for an appropriate consideration of the opportu- nity costs of the coverage decision [21]. Instead, costs can effect’ were consistently assigned the highest scores by the three groups. This means that all groups believed that be considered separately to explicitly trade-off (incre- iNHL is a severe condition, given the patients’ life mental) benefits generated by a new treatment against its expectancy after diagnosis and possible persistence of (incremental) costs (for a discussion about this issue see symptoms, and that obinutuzumab could bring clinical Garau and Devlin [22]). When decision makers face a fixed benefits at the patient level, including moderately budget constraint, an aggregate measure of benefit (similar delaying progression and helping to control disease to the score presented in Fig. 5) can be compared with an symptoms. Both payers and patients thought that another estimate of costs. This approach is presented by Golan and area where obinutuzumab could bring value is ‘unmet Hansen, who developed an MCDA framework piloted by needs’. Available interventions for iNHL have limitations the Israeli Advisory Committee to select new interventions (e.g. a proportion of the population does not respond to to fund [23]. SoC), which need to be addressed. Clinicians thought Figure 4 also shows that the key drivers of the obinu- that data presented was relevant to decision makers and tuzumab score value are ‘disease severity’ (which accounts valid with respect to scientific standard. Finally, in the for approximately 18% of the total value), ‘type of thera- economic-related criteria ‘comparative cost of interven- peutic benefit’ and ‘unmet needs’ (which accounts for tion’ and ‘comparative other medical costs’, obinu- approximately 13% of the total value). tuzumab obtained a negative score (between -2.8 and We conducted a sensitivity analysis where we set the 0.4) when compared with its comparator bendamustine, weight for the comparative costs criteria to zero, scaled up Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 159 Fig. 4 Overall value score for obinutuzumab (all stakeholder groups combined) and contribution of each criterion to the total value Fig. 5 ‘‘Benefit’’ score for obinutuzumab (sensitivity analysis) and contribution of each criterion to total benefit. Note The benefit score is the overall value score excluding the criteria relating to cost the weights for the remaining criteria clusters, and recal- benefit assessment, similar to that followed by the AIFA in culated the overall score from a combined perspective. The Italy. result is shown in Fig. 5. Removing the cost criteria increases the obinutuzumab value score from 0.45 to 0.55. If this version of the 4 Discussion framework was used, decision makers would need to assess and consider the net economic impact alongside this benefit In many systems, including the Italian system, the per- score. This approach might also be helpful in those systems spectives of stakeholders such as patients are not elicited or where the price of the intervention is defined following its incorporated at any stage of the assessment and decision- 160 M. Garau et al. making process. The use of an MCDA framework such as Currently, the AIFA does consider some of the criteria EVIDEM could enable the collection of stakeholders’ included in the EVIDEM framework; however, this is not preferences (via weights allocation) and help to ensure that done systematically and it remains unclear how evidence they are taken into account more systematically in decision on those criteria is developed and to what extent it influ- making (via determination of the value score and its con- ences decision making. An MCDA process such as the one sideration in decision making). applied in this study can make both aspects more explicit The value score can help identify the key criteria and lead to more consistent consideration of multiple cri- impacting the intervention’s value, and lead to an in-depth teria in decision making. To implement MCDA in practice discussion within the decision-making committee around at national level, broader and larger groups of stakeholders the evidence presented on those criteria and the level of embracing different disease areas would need to be con- consensus that was obtained across participants when sulted. Alternatively, the decision-making committee (ei- assigning weights and scores. It can also inform sensitivity ther the Technical Scientific Committee or the Prices and analyses evaluating the robustness of the decision outcome. Reimbursement Committee within the AIFA) can act as the The value score has limited use in absolute terms if there agent and represent different stakeholders (such as the local is no specification on how it should inform coverage NHS payers, general public and patients—the principals) in decisions. If used to compare and rank competing tech- determining the relative importance of criteria. nologies, or across successive decisions about different technologies, score values might be useful. An example of this approach is the MCDA framework for health tech- 5 Limitations nology prioritisation developed for the Israeli Advisory Committee [23]. This approach used the benefit score and This study was exploratory as it applied MCDA in the the net total cost to draw efficiency frontiers and, based on context of medicine reimbursement decision making in budget constraints, selected the technologies to be funded. Italy in a convenience sample. In future uses, improve- On the other hand, for repeated reimbursement or HTA ments can be made to increase the validity of results and decisions affecting a fixed budget, there is a need to define their applicability in formal decision-making processes. the ‘hurdle for adoption’ [24]; in other words, the incre- Our convenience sample ensured a high rate of response mental cost per value score to compare against the cost per but its size and clinical areas covered could be expanded, value score of individual interventions to understand e.g. involving patients and clinicians of other non-oncology whether they are good value for money. However, given conditions. Survey instructions and synthesis of evidence the methodological issues in defining and estimating the on obinutuzumab were provided in English, while during opportunity cost of HTA decisions [25–28] and the role of the meetings, the participants and the moderator spoke in regional (as opposed to national) jurisdictions in the man- Italian, which helped in the interpretation of the scientific agement of the health budget, there might be a need to evidence and the instructions. If the exercise is conducted develop new approaches to ensure efficient decision mak- on a larger scale, it would benefit from the translation of all ing [22]. the material into the relevant language to increase under- In the context of the Italian National Health Service standing and rate of responses. (NHS), EVIDEM is implemented in the Lombardia region Patient representatives also raised the need to simplify to make listing and de-listing decisions on medical devices the language used to explain the framework and make it [7]. More than 20 interventions have been appraised and more accessible to lay persons. This shows that the inter- have obtained value scores, including economic criteria, active component is not only needed at the end of the between 0.22 and 0.72 [9]. Therefore, obinutuzumab fits in process, to consolidate survey responses, but also at the the middle of this range. However, the version of EVIDEM start, to ensure full understanding around the criteria and used in Lombardia is slightly different to that used in our their definitions. Challenges in communicating the elicita- study; it is applied in a regional context rather than a tion exercise to patients was also highlighted by Marsh national context, and information on which scores, on et al. [30]. Therefore, validation with stakeholders should average, allowed interventions to be approved for reim- be part of each step of an MCDA exercise [21]. bursement does not appear to be available. Given the importance of the interaction part of an We should also highlight that, consistent with the pur- MCDA process, future exercise should include qualitative poses of MCDA, value scores are intended to inform and analysis on methods used to structure group discussions support decision making and not to be used as a prescriptive and the type of interaction. Exploring how structured group rule in place of deliberations. A deliberative component is discussions change (or not) when participants meet in seen as necessary in all decision-making processes [29]. person compared with when they interact in a virtual space Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 161 (e.g. webinar) might help to establish efficient and effective domains and secondly among criteria within each domain, ways to support group dynamics. we noted some distortions for the values obtained by cri- In addition, as discussed in Mu ¨ hlbacher and Kaczynski teria in domains with two criteria, which tended to have [31], there are underlying difficulties in understanding the higher weights, versus criteria in domains with three cri- data provided and interpreting the score scales. For teria. One way to avoid this is to assign points directly to example, three patients reported that obinutuzumab was the criteria rather than splitting the task into two stages. For cheaper than the comparator (in the cost of intervention larger-scale applications of MCDA, alternative instruments criterion), which is not in line with what we would have to elicit preferences that have strong theoretical founda- expected from the data presented to them. This may reveal tions, and that have been used in other types of healthcare that they did not fully understand the task. In contrast, all assessments, can be considered, such as discrete choice patients reported that obinutuzumab was more clinically experiments and PAPRIKA [33]. effective than the comparator, which is in line with what we would have expected. The interval scale incorporated into EVIDEM has been 6 Conclusions tested and validated in several applications [31] and refined accordingly over time. For example, scores were initially To our knowledge, this is the first time that an MCDA measured on a scale from 0 to 3 [13] and subsequently on approach has been used to inform reimbursement decision scale from 0 to 5, probably to increase its discriminatory making of medicines at the national level in Italy. power. Others have suggested the use of a scale from 0 to This study showed that MCDA (in particular, EVIDEM) 100 [32]. Further validation exercises might be required to can be used to elicit the views of different stakeholder assess consistency and interval properties of the scoring groups. We found that the views of patients and clinicians scale. were broadly aligned as they expressed preference for We observe that more guidance needs to be provided in interventions targeting severe conditions and they ranked order to score some of the criteria, particularly the eco- economic criteria as the least important interventions. nomic-related criteria. What constitutes ‘substantial addi- Payers allocated higher weight (compared with patients tional expenditures’ (corresponding to a score of -5) for and clinicians) to the economic criteria and to the quality of one respondent might be different to that of another evidence. The key criteria driving the value of obinu- respondent. We included information about the national tuzumab accordingly to all the stakeholder groups were pharmaceutical expenditure to provide some context; disease severity, type of therapeutic benefit and unmet however, clear ranges or cut-off values should be included needs. for each score to ensure consistency in responses. Decision makers in Italy already consider some of the The validation process should also ensure that criteria EVIDEM criteria, such as disease severity, but with no are ‘preference independent’, meaning that it is possible to systematic approach. The perspectives of stakeholders judge how well one criterion is achieved without knowing (such as patients) are not elicited or incorporated at any how well any of the other criteria are achieved. One par- stage of the assessment or decision-making process. Our ticipant pointed out that the criterion ‘unmet needs’ should MCDA study provides useful evidence to decision makers, be considered in conjunction with ‘disease severity’ as the such as the AIFA, on what health interventions attribute lack of alternative interventions is meaningless to decision different stakeholders’ value the most, and has tested makers if it is not referred to a serious condition. This issue methods to ensure that this is captured consistently across can be addressed with alternative aggregation approaches, different interventions. such as multiplicative instead of additive methods [21]. To limit the cognitive burden of participants, we focused the meeting discussions on a restricted number of criteria Author contributions MG and ND designed the study. MG con- (the 13 EVIDEM ‘core criteria’). There is no rule on the ducted the structured meetings, with input from NAM and AP. GH set up the survey, analysed the data in Microsoft Excel, and reviewed the optimal number of criteria to include in an MCDA evidence included in the product evidence synthesis report. All framework; however, it is important to consider the trade- authors contributed to the drafting of the manuscript. off between breath/inclusiveness of a framework and resources needed to develop relevant evidence and analyse Compliance with Ethical Standards it for decision making [21]. Funding The work on this paper was funded by Roche S.P.A. with On the method to obtain weights, we selected one of the the intention of publishing the results from the outset. methods recommended by the EVIDEM collaboration: the point allocation approach. However, because of the top- Conflict of interest Martina Garau, Grace Hampson and Nancy Devlin are employed by The Office of Health Economics, which down approach used to allocate points, firstly among the 162 M. Garau et al. receives funding from a variety of sources, including the Association context-specific adaptation. Pharmacoeconomics. of the British Pharmaceutical Industry. Nicola Amedeo Mazzanti and 2016;34(3):285–301. Antonio Profico are employees of Roche S.P.A. 12. Daniels N, Sabin JE. Accountability for reasonableness: an update. BMJ. 2008;337:a1850. 13. Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P, Data availability statement Our input data and the material circu- et al. Bridging health technology assessment (HTA) with multi- lated to the study participants is available as electronic supplementary criteria decision analyses (MCDA): field testing of the EVIDEM material. framework for coverage decisions by a public payer in Canada. BMC Health Serv Res. 2011;11:329. Open Access This article is distributed under the terms of the 14. Gilabert-Perramon A, Catalan A, Prat A, Fontanet M, Puig-Peiro ´ Creative Commons Attribution-NonCommercial 4.0 International R, Merino-Montero S, et al. Development of a multi-criteria License (http://creativecommons.org/licenses/by-nc/4.0/), which per- decision analysis (MCDA) framework for health care decision- mits any noncommercial use, distribution, and reproduction in any making in Catalonia (Spain): pilot study focused in orphan drugs. medium, provided you give appropriate credit to the original Vienna: ISPOR 19th Annual European Congress; 2016. author(s) and the source, provide a link to the Creative Commons 15. van Til J, Groothuis-Oudshoorn C, Lieferink M, Dolan J, Goet- license, and indicate if changes were made. ghebeur M. Does technique matter: a pilot study exploring weighting techniques for a multi-criteria decision support framework. Cost Eff Resour Alloc. 2014;12:22. 16. Sehn LH, Chua N, Mayer J, Dueck G, Trneny M, Bouabdallah K, et al. Obinutuzumab plus bendamustine versus bendamustine References monotherapy in patients with rituximab-refractory indolent non- Hodgkin lymphoma (GADOLIN): a randomised, controlled, 1. HTAi Patient and Citizen Involvement. Good Practice Examples open-label, multicentre, phase 3 trial. Lancet Oncol. of Patient and Public Involvement in Health Technology 2016;17(8):1081–93. Assessment. 2015. http://www.htai.org/fileadmin/HTAi_Files/ 17. European Medicines Agency. Assessment report Gazyvaro Procedure ISG/PatientInvolvement/EffectiveInvolvement/Good_Practice_ No. EMEA/H/C/002799/II/0007. 2016. http://www.ema.europa.eu/ Examples_Feb_2015.pdf. Accessed 14 Aug 2017. docs/en_GB/document_library/EPAR_-_Assessment_Report_-_Vari 2. SMC. PACE—patient and clinician engagement. Factsheet. 2015. ation/human/002799/WC500211313.pdf. Accessed 14 Aug 2017. https://www.scottishmedicines.org.uk/files/PACE/PACE_facts 18. Lee L, Wang L, Crump M. Identification of potential surrogate heet_0417.pdf. Accessed 14 Aug 2017. end points in randomized clinical trials of aggressive and indolent non-Hodgkin’s lymphoma: correlation of complete response, 3. Shah KK, Mestre-Ferrandiz J, Towse A, Smyth EN. A review of time-to-event and overall survival end points. Ann Oncol. health technology appraisals: case studies in oncology. Int J 2011;22(6):1392–403. Technol Assess Health Care. 2013;29(1):101–9. 19. Belton V, Pictet J. A framework for group decision using a 4. Devlin N, Sussex J. Incorporating multiple criteria in HTA: MCDA model: sharing, aggregating or comparing individual methods and processes. OHE Monograph. London: Office of information? J Decis Syst. 1997;6(3):283–303. Health Economics; 2011. 20. Claxton K. Three questions to ask when examining MCDA. 5. Danner M, Hummel JM, Volz F, van Manen JG, Wiegard B, Value Outcomes Spotlight. 2015;1:10–8. Dintsios CM, et al. Integrating patients’ views into health tech- 21. Marsh K, IJmerzan M, Thokala P, Baltussen R, Boysen M, Kalo nology assessment: analytic hierarchy process (AHP) as a method Z, et al. Multiple criteria decision analysis for health care deci- to elicit patient preferences. Int J Technol Assess Health Care. sion making—emerging good practices: report 2 of the ISPOR 2011;27(4):369–75. MCDA emerging good practices task force. Value Health. 6. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, 2016;19(2):125–37. et al. Multiple criteria decision analysis for health care decision making-an introduction: report 1 of the ISPOR MCDA emerging 22. Garau M, Devlin NJ. Using MCDA as a decision aid in health good practices task force. Value Health. 2016;19(1):1–13. technology appraisal for coverage decisions: opportunities, 7. Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali challenges and unresolved questions. In: Marsh K, Goetghebeur M. Implementation of EUnetHTA core Model(R) in Lombardia: M, Thokala P, Baltussen R, editors. Multi-criteria decision the VTS framework. Int J Technol Assess Health Care. analysis to support healthcare decisions. Cham: Springer Inter- 2014;30(1):105–12. national Publishing; 2017. p. 277–98. 8. Castro H, Tringali M, Cleemput I, Devriese S, Leoni O, Lettieri 23. Golan O, Hansen P. Which health technologies should be funded? E. Advancing MCDA and HTA into coverage decision-making. A prioritization framework based explicitly on value for money. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, editors. Isr J Health Policy Res. 2012;1(1):44. Multi-criteria decision analysis to support healthcare decisions. 24. Sussex J, Towse A, Devlin N. Operationalizing value-based Cham: Springer International Publishing; 2017. p. 119–46. pricing of medicines: a taxonomy of approaches. Pharmacoeco- 9. Wahlster P, Goetghebeur M, Schaller S, Kriza C, Kolominsky- nomics. 2013;31(1):1–10. Rabas P. Exploring the perspectives and preferences for HTA 25. Culyer A, McCabe C, Briggs A, Claxton K, Buxton M, Akehurst across German healthcare stakeholders using a multi-criteria R, et al. Searching for a threshold, not setting one: the role of the assessment of a pulmonary heart sensor as a case study. Health National Institute for Health and Clinical Excellence. J Health Serv Res Policy. 2007;12(1):56–8. Res Policy Syst. 2015;13:24. 26. Karlsberg Schaffer S, Sussex J, Devlin N, Walker A. Local health 10. Bentley TG, Cohen JT, Elkin EB, Huynh J, Mukherjea A, Neville care expenditure plans and their opportunity costs. Health Policy. TH, et al. Validity and reliability of value assessment frameworks 2015;119(9):1237–44. for new cancer drugs. Value Health. 2017;20(2):200–5. 27. Appleby J, Devlin N, Parkin D, Buxton M, Chalkidou K. 11. Wagner M, Khoury H, Willet J, Rindress D, Goetghebeur M. Can Searching for cost effectiveness thresholds in the NHS. Health the EVIDEM framework tackle issues raised by evaluating Policy. 2009;91(3):239–45. treatments for rare diseases: analysis of issues and policies, and Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 163 28. Claxton K, Martin S, Soares M, Rice N, Spackman E, Hinde S, research and future development of MCDA. Appl Health Econ et al. Methods for the estimation of the National Institute for Health Policy. 2016;14(1):29–40. Health and Care Excellence cost-effectiveness threshold. Health 32. Phillips LD. Best Practice for MCDA in Healthcare. In: Marsh K, Technol Assess. 2015;19(14):1–503 (v–vi). Goetghebeur M, Thokala P, Baltussen R, editors. Multi-criteria 29. Baltussen R. Question is not whether but how to use MCDA. decision analysis to support healthcare decisions. Cham: Springer Value Outcomes Spotlight. 2015;1(1):14–6. International Publishing; 2017. p. 311–29. 30. Marsh K, Caro JJ, Hamed A, Zaiser E. Amplifying each patient’s 33. Golan O, Hansen P, Kaplan G, Tal O. Health technology prior- voice: a systematic review of multi-criteria decision analyses itization: which criteria for prioritizing new technologies and involving patients. Appl Health Econ Health Policy. what are their relative weights? Health Policy. 2017;15(2):155–62. 2011;102(2–3):126–35. 31. Mu ¨ hlbacher AC, Kaczynski A. Making good decisions in healthcare with multi-criteria decision analysis: the use, current http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png PharmacoEconomics - Open Springer Journals

Applying a Multicriteria Decision Analysis (MCDA) Approach to Elicit Stakeholders’ Preferences in Italy: The Case of Obinutuzumab for Rituximab-Refractory Indolent Non-Hodgkin Lymphoma (iNHL)

Free
11 pages
Loading next page...
 
/lp/springer_journal/applying-a-multicriteria-decision-analysis-mcda-approach-to-elicit-IhVgQWsPzq
Publisher
Springer International Publishing
Copyright
Copyright © 2017 by The Author(s)
Subject
Medicine & Public Health; Pharmacoeconomics and Health Outcomes
ISSN
2509-4262
eISSN
2509-4254
D.O.I.
10.1007/s41669-017-0048-x
Publisher site
See Article on Publisher Site

Abstract

PharmacoEconomics Open (2018) 2:153–163 https://doi.org/10.1007/s41669-017-0048-x OR IGINAL RESEARCH ARTIC L E Applying a Multicriteria Decision Analysis (MCDA) Approach to Elicit Stakeholders’ Preferences in Italy: The Case of Obinutuzumab for Rituximab-Refractory Indolent Non-Hodgkin Lymphoma (iNHL) 1 1 1 2 • • • • Martina Garau Grace Hampson Nancy Devlin Nicola Amedeo Mazzanti Antonio Profico Published online: 22 August 2017 The Author(s) 2017. This article is an open access publication Abstract Conclusions MCDA can be used to elicit the views of Background Healthcare decision makers need to make different stakeholder groups and has the potential to trade-offs between different elements of value of new structure and inform reimbursement decisions. treatments. Multicriteria decision analysis (MCDA) pro- vides a framework that can help decision makers to understand stakeholders’ preferences and be explicit about Key Points for Decision Makers the trade-offs that are being made. Objective The objective of this study was to use MCDA to Multicriteria decision analysis (MCDA) provides a obtain preferences and views on decision criteria across framework that can help decision makers to three stakeholder groups (patients, clinicians and payers) in understand stakeholders’ preferences and be explicit Italy and to use these to assess the performance of obinu- about the trade-offs that are being made between tuzumab for rituximab-refractory indolent non-Hodgkin different elements of value. lymphoma (iNHL). Based on a convenience sample, we found that Methods We used EVIDEM V3.0, an MCDA framework, patients and clinicians give greater weight to and collected participants’ preferences via an online survey targeting severe conditions; payers are most and structured meetings. concerned with unmet need, comparative costs, and Results Patients and clinicians expressed a preference for high-quality evidence. interventions targeting severe conditions. Payers expressed preference for treatments targeting areas with an unmet The overall value score of obinutuzumab based on need, which are cheaper than the comparator, and with all stakeholder groups’ responses combined was high-quality evidence. Obinutuzumab in combination with mainly driven by the criteria disease severity, type of bendamustine, compared with bendamustine alone, therapeutic benefit and unmet needs. received high positive scores for the criteria ‘disease severity’ and ‘type of therapeutic benefit’ by all three groups, and negative scores on the economic-related cri- teria, according to all stakeholder groups. 1 Background Electronic supplementary material The online version of this article (doi:10.1007/s41669-017-0048-x) contains supplementary material, which is available to authorized users. Stakeholders such as patients can be involved in medicine reimbursement decisions and Health Technology Assess- & Martina Garau ment (HTA) in different ways, from contributions to evi- mgarau@ohe.org dence submissions to participation in advisory committees. Office of Health Economics, London, UK In some HTA processes, such as that of the National Institute for Health and Care Excellence (NICE) in Roche S.P.A, Milan, Italy 154 M. Garau et al. England and Wales, and the Canadian Agency for Drugs 2 Methods and Technologies in Health (CADTH) in Canada, patients submit evidence and participate in committee meetings [1]. There are number of approaches to eliciting preferences The Scottish Medicines Consortium (SMC) in Scotland has with different levels of complexity and theoretical bases, introduced the option of consulting a panel of clinicians including swing weights, analytic hierarchy process and and patient representatives during the assessment of discrete choice experiments [6]. Recently, there has also medicines for rare or end-of-life conditions [2]. been a proliferation of frameworks to assess the value of HTA decision makers need to understand how stake- new interventions, developed by organisations including holders trade-off between different types of benefit poten- the American Society of Clinical Oncology (ASCO), the tially generated by a new treatment. Stakeholders need to European Society for Medical Oncology (ESMO) and the understand to what extent and how their input affects final Institute for Clinical and Economic Review (ICER) [10]. HTA decisions. A study of appraisals in five countries [3] For the purposes of this study, we selected the EVIDEM found that patient preferences are rarely mentioned in HTA approach, which combines a coherent set of criteria with reports, and that patient participation in HTA tend to be tools to operationalise it in decision making (including unsystematic. suggested approaches for preference elicitation). In addi- Multicriteria decision analysis (MCDA) provides a clear tion, EVIDEM was specifically designed for healthcare framework to assess the value of a treatment compared decision making. Its choice here enables results to be with alternative treatments or standard of care (SoC), compared with a growing literature from its use in other against multiple and competing criteria [4]. It can support healthcare decision-making settings. decision makers to be explicit about the trade-offs made EVIDEM (V3.0) is an open-source framework resulting between the selected criteria, and offers systematic and from a collaboration of experts and stakeholders, and is robust ways to elicit preferences and consider evidence subject to continued testing and adaptations [11]. In the last from stakeholders. For example, in Germany, the Institute 10 years, the EVIDEM collaboration has merged the for Quality and Efficiency in Health Care (IQWiG) ran practical aspects of developing an implementable frame- MCDA pilots to elicit patient preferences on clinical out- work with ethical foundations for its criteria and objectives. comes in depression and hepatitis C [5, 6]. In Italy, the In particular, ethical justification has been provided to the Lombardia region uses an MCDA framework to select criteria, including ‘‘distributive justice and fairness (pri- health technologies to reimburse [7]. oritise those who are worst off)’’ [11]. EVIDEM also The purpose of this study was to use an MCDA advocates for important procedural values such as ‘partic- approach to obtain preferences and views on criteria and on ipatory decision making’, which aligns with the ethical performance of obinutuzumab for rituximab-refractory framework of accountability for reasonableness developed indolent non-Hodgkin lymphoma (iNHL) across three by Daniels and Sabin for ‘legitimate’ priority settings stakeholder groups (patients, clinicians and payers) in Italy. [11, 12]. As far we know, this is the first study exploring stake- EVIDEM comprises a broad range of criteria, capturing holders’ preferences to inform national reimbursement elements of value relevant to patients, healthcare systems decisions of medicines in Italy. While Castro et al. describe and society, with properties (including non-overlap the use of EVIDEM in Italy in the context of a regional between criteria), and with operational purposes, enabling payer and focused on medical devices, we conducted our us to inform real-life decision making. EVIDEM has been MCDA exercise to inform value assessment and decision applied in HTA in a number of jurisdictions, including making at the national level, specifically in the context of Canada [13] and, more recently, Spain, where the HTA medicines [8], which is currently under the responsibility body in Catalonia has explored the framework for of Agenzia Italiana del Farmaco (AIFA ). appraising orphan drugs [14]. The Italian region Lombardia Our work built on the study by Wahlster et al. [9], who uses EVIDEM to inform local funding decisions of health applied EVIDEM to capture stakeholders’ preferences in interventions. Our aim was to extend the application of Germany using an intervention case study. In addition to MCDA to support national decision making of medicine the online survey that was used to obtain preferences reimbursement. individually by Wahlster et al., we applied EVIDEM to EVIDEM includes a set of 13 clearly defined and support group discussions. This is a fundamental role that measurable ‘core’ criteria grouped into domains. Some MCDA can play in HTA to structure committees’ consid- criteria are measured in absolute terms, not relative to other eration of different and often conflicting perspectives. interventions, and other criteria are measured compara- tively to existing interventions. 1 2 http://www.agenziafarmaco.gov.it/en. https://www.evidem.org/. Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 155 Figure 1 presents the core EVIDEM framework, because it combines simplicity with the ability of ‘forcing’ including five domains: need for intervention, comparative people to prioritise criteria. We also noted that van Til et al. outcomes of intervention, type of benefit, economic con- [15] showed that the choice of the weight elicitation sequences of intervention, and knowledge about interven- method does not affect value estimates at the group level. tion. Each domain comprises a set of criteria. The description of the criteria and the survey instructions In May 2016, an online survey (using SurveyMonkey ) given to participants are available in the electronic sup- was sent to three groups of stakeholders involved or plementary material. To obtain one set of weights for each affected by reimbursement decisions in Italy: clinicians, criterion, we combined domain weights with those within patients, and payers. This step of the process was labelled each domain and normalised the values (to sum up to 1). as the ‘survey first round’. The survey was used to elicit Respondents were asked to score the performance of preferences around the relative importance of criteria obinutuzumab in combination with bendamustine followed (weights), and the degree of achievement of obinutuzumab by obinutuzumab maintenance, compared with ben- against these criteria compared with the current SoC damustine alone (which is the only efficacious intervention (scores). in this indication [16]), in patients with rituximab-refrac- We derived weights using the ‘point allocation’ tory iNHL. This is in line with obinutuzumab license approach, where we asked participants to allocate 100 indication and clinical evidence [16, 17]. points, first across criterion domains and, second, across Incremental criteria, related to the health and non-health criteria within each clusters. This method was selected effects of the intervention, were measured on a scale from Fig. 1 Structure of the MCDA EVIDEM framework, including all criteria and categories 156 M. Garau et al. -5to ?5. Absolute criteria were measured on a scale from Participants were drawn from existing manufacturer 0to5. networks. The payer group involved hospital, regional and Evidence on obinutuzumab for iNHL was based on lit- national decision makers; patient representatives were erature reviews included in HTA submissions and results of members of Italian patient groups related to lymphomas; obinutuzumab clinical trials. Evidence was reviewed and and clinicians were lymphoma specialists. Members of synthesised using the EVIDEM framework. The evidence each group were distributed across Italian regions, covering matrix provided to participants is presented in the elec- the north, centre, and south areas. tronic supplementary material. Most of the evidence for the ‘comparative efficacy/ef- fectiveness’ criterion referred to one of the clinical end- 3 Results points measured in the clinical trial, i.e. progression-free survival [16]. We recognise that this is a surrogate endpoint A total of 19 people were invited and completed the first with different levels of associations with overall survival round of the survey, including nine patients, five clinicians (OS), which is the primary measure of efficacy [18]. and five payers. The structured meetings were found to be Nevertheless, there is an increasing acceptance of surrogate useful by participants to discuss the MCDA framework and endpoints by regulatory agency and other healthcare deci- study approach with their peers. Only the clinician group sion makers, given the additional time and resources reviewed their answers following the structured meetings. required for collecting evidence to estimate OS. Responses were analysed in Microsoft Excel (Microsoft 3.1 Stakeholders’ Weights Corporation, Redmond, WA, USA), and key results, including average and standard deviation (SD) of weights Weights represent trade-offs between criteria [6] and thus and score values from the survey first round, were pre- reveal which aspects of value matter most to each group. sented at structured meetings. Structured meetings, one To obtain ‘generic’ weights to apply to any intervention, with each stakeholder group, were run to allow participants respondents were required to express their preferences to seek further explanation on the MCDA framework, to between the EVIDEM criteria, based solely on their defi- discuss and review weights and scores obtained, identify nition, and scale measurement, not the description of the areas of agreement and disagreement among participants, intervention. and, where possible, reach a consensus on the weight and Figure 2 compares the sets of normalised weights from score values that could best represent the group’s each group. According to patients, the two most important perspective. criteria were the ‘type of therapeutic benefit’ and ‘disease To minimise the participants’ cognitive burden and have severity’, both with weights of 11% (SD 0.07 and 0.10, a manageable number of criteria, the meeting discussions respectively). These are both absolute criteria (not relative concentrated on the EVIDEM core criteria (presented in to comparative interventions). The three least important Fig. 1) and did not include the contextual qualitative cri- criteria were the three economic indicators: ‘comparative teria, which were omitted from the final results. The con- non-medical costs’, ‘comparative other medical costs’, and textual criteria do not require weights and scores, therefore ‘comparative cost of intervention’, with weights of 3% (SD their exclusion do not affect the overall value score. 0.03), 4% (SD 0.04) and 5% (SD 0.04), respectively. Participants were invited to complete the survey in light According to clinicians, the two most important criteria of the discussion at the meeting (survey second round). were ‘disease severity’ and ‘comparative efficacy/effec- Following this, the average of the normalised weights and tiveness’, with weights of 15% (SD 0.10) and 12% (SD scores from the three groups were combined with linear 0.05), respectively. The three least important criteria were aggregation to calculate the intervention value score. The two economic indicators (‘comparative non-medical costs’ literature suggests a variety of approaches to aggregating and ‘comparative other medical costs’), and ‘type of pre- the preferences and views expressed by individuals to ventative benefit’, with weights of 3% (SD 0.02), 4% (SD inform group decision making. These include agreeing the 0.02) and 4% (SD 0.03), respectively. weight and score values as part of the committee discus- Finally, payers indicated that the three most important sion; aggregating by using the average of weights of scores criteria were ‘unmet needs’, ‘comparative cost of inter- obtained from responders; and retaining and comparing vention’, and ‘quality of evidence’, with weights of 11% respondents’ values [19]. We implemented the third (SD 0.07), 11% (SD 0.05) and 10% (SD 0.02), respec- approach as we observed the differences between groups’ tively. The three least important criteria were ‘comparative and individuals’ values. We also used the second approach non-medical costs’, ‘size of affected population’ and as an example of incorporating stakeholders’ preferences in ‘comparative other medical costs’, with scores of 4% (SD value assessment. 0.01), 5% (SD 0.02) and 5% (SD 0.02), respectively. The Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 157 Fig. 2 Relative weights of Type of therapeuc benefit individual criteria by Disease severity stakeholder group Comparave paent-received health/PRO Comparave effecveness Type of prevenve benefit Expert consensus/clinical pracce guidelines Comparave safety/tolerability Unmet needs Size of affected populaon Quality of evidence Comparave cost consequences - cost of intervenon Comparave cost consequences - other medical costs Comparave cost consequences - non-medical costs 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 Mean normalized weights Clinician Payer Paent small SDs reflect a high level of agreement within this quality evidence. In contrast, the quality of evidence cri- stakeholder group about the importance of each criteria, terion was deemed a low priority by patients. despite a small sample size. Given the limited sample size, we did not perform 3.2 Stakeholders’ Scores any statistical comparison across stakeholder group weights; however, it is worth highlighting key differ- Based on different outcome measures and types of evi- ences and commonalities. Compared with patients and dence on obinutuzumab for the treatment of rituximab-re- clinicians, payers distributed the weights more equally fractory iNHL, participants provided scores. Unlike among the domains. Payers’ weights range between 17 weights, scores are specific to the intervention under con- and 24%, compared with those given by clinicians and sideration. Figure 3 presents the scores allocated by the patients, which range between 12 and 33%. These two three stakeholder groups. groups give less weight to the domains ‘economic con- Patients assigned the highest scores (representing the sequences of intervention’ and ‘knowledge about the areas where obinutuzumab performs best) to ‘unmet intervention’. needs’, with an average score (AS) of 3.7 (SD 0.65); Patients’ and clinicians’ views were aligned as they ‘disease severity’, with an AS of 3.6 (SD 0.46); and ‘type expressed preference for interventions targeting severe of therapeutic effect’, with an AS of 3.1 (SD 1.05). The conditions. The highest weights in both group were criteria part of the ‘comparative outcomes of intervention’ observed for this criterion. Patients also believed that pri- domain were, on average, all positive, indicating that ority should be given to interventions that have a signifi- obinutuzumab is expected to generate incremental health cant therapeutic effect (for example, they offer a cure or gains compared with SoC. However, in the criterion ‘pa- significantly delay progression of the disease), while clin- tient-perceived health/patient-reported outcomes’, there icians indicated that one of the most important criteria is was a large variation in the assigned scores, which ranged the improvement in clinical outcomes compared with SoC. from -2to ?5. All three criteria related to the economic Both groups ranked all the economic criteria among the impact of implementing the intervention were scored five least important criteria. negative, on average. Payers allocated higher weights to the economic-related Areas in which obinutuzumab was deemed to perform criteria, with the direct (incremental) cost of the interven- better than the comparator by clinicians were ‘size of tion being one of the most important. Their preferences affected population’, with an AS of 4 (SD 0.71); and were for treatments targeting populations in which there is ‘disease severity’, ‘type of therapeutic effect’, and ‘quality little or no effective treatment, which are less expensive of evidence’, with an AS of 3.6 (SD 0.89, 0.89, 0.55, than the comparator, and which are underpinned by high- respectively). Two criteria part of the ‘comparative 158 M. Garau et al. Fig. 3 Mean scores for Scores for obinutuzumab for the three stakeholder groups obinutuzumab for each criteria Type of therapeuc benefit by stakeholder group Disease severity Comparave paent-received health/PRO Comparave effecveness Type of prevenve benefit Expert consensus/clinical pracce guidelines Comparave safety/tolerability Unmet needs Size of affected populaon Quality of evidence Comparave cost consequences - cost of intervenon Comparave cost consequences - other medical costs Comparave cost consequences - non-medical costs -4 -3 -2 -1 0123 4 Clinician Payer Paent outcomes of intervention’ domain were, on average, posi- whose patent has recently expired, according to all tive. However, in terms of safety and tolerability, obinu- stakeholder groups. This is because the cost of obinu- tuzumab was deemed slightly worse than its comparator tuzumab in combination with bendamustine and related [with a score of -0.4 (SD 0.89)]. Clinicians also assigned medical costs was higher than those of bendamustine negative scores to obinutuzumab against ‘comparative cost alone. of intervention’ and ‘comparative other medical costs’, which obtained a score of -1.2 (SD 1.10) and -0.4 (SD 3.3 Overall Value Score of Obinutuzumab 0.89), respectively. Payers gave the highest scores to ‘disease severity’ and To develop a combined perspective on obinutuzumab ‘unmet needs’, with an AS of 4.0 (SD 0.71, 0.71, respec- value, all survey responses were included and weighted tively); and ‘type of therapeutic benefit’, with an AS of 3.4 equally. As shown in Fig. 4, the value score of obinu- (SD 0.55). Two criteria received negative scores, on tuzumab was 0.45. average: ‘comparative cost of intervention’ and ‘compar- A number of MCDA best-practice articles challenge the ative other medical costs’. inclusion of (incremental) costs as a separate criterion. If We observed some consistency across stakeholder the overall score is a composite measure of benefit, costs groups in relation to obinutuzumab scores. are not an attribute of benefit [20]. In addition, this would The criteria ‘disease severity’ and ‘type of therapeutic not allow for an appropriate consideration of the opportu- nity costs of the coverage decision [21]. Instead, costs can effect’ were consistently assigned the highest scores by the three groups. This means that all groups believed that be considered separately to explicitly trade-off (incre- iNHL is a severe condition, given the patients’ life mental) benefits generated by a new treatment against its expectancy after diagnosis and possible persistence of (incremental) costs (for a discussion about this issue see symptoms, and that obinutuzumab could bring clinical Garau and Devlin [22]). When decision makers face a fixed benefits at the patient level, including moderately budget constraint, an aggregate measure of benefit (similar delaying progression and helping to control disease to the score presented in Fig. 5) can be compared with an symptoms. Both payers and patients thought that another estimate of costs. This approach is presented by Golan and area where obinutuzumab could bring value is ‘unmet Hansen, who developed an MCDA framework piloted by needs’. Available interventions for iNHL have limitations the Israeli Advisory Committee to select new interventions (e.g. a proportion of the population does not respond to to fund [23]. SoC), which need to be addressed. Clinicians thought Figure 4 also shows that the key drivers of the obinu- that data presented was relevant to decision makers and tuzumab score value are ‘disease severity’ (which accounts valid with respect to scientific standard. Finally, in the for approximately 18% of the total value), ‘type of thera- economic-related criteria ‘comparative cost of interven- peutic benefit’ and ‘unmet needs’ (which accounts for tion’ and ‘comparative other medical costs’, obinu- approximately 13% of the total value). tuzumab obtained a negative score (between -2.8 and We conducted a sensitivity analysis where we set the 0.4) when compared with its comparator bendamustine, weight for the comparative costs criteria to zero, scaled up Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 159 Fig. 4 Overall value score for obinutuzumab (all stakeholder groups combined) and contribution of each criterion to the total value Fig. 5 ‘‘Benefit’’ score for obinutuzumab (sensitivity analysis) and contribution of each criterion to total benefit. Note The benefit score is the overall value score excluding the criteria relating to cost the weights for the remaining criteria clusters, and recal- benefit assessment, similar to that followed by the AIFA in culated the overall score from a combined perspective. The Italy. result is shown in Fig. 5. Removing the cost criteria increases the obinutuzumab value score from 0.45 to 0.55. If this version of the 4 Discussion framework was used, decision makers would need to assess and consider the net economic impact alongside this benefit In many systems, including the Italian system, the per- score. This approach might also be helpful in those systems spectives of stakeholders such as patients are not elicited or where the price of the intervention is defined following its incorporated at any stage of the assessment and decision- 160 M. Garau et al. making process. The use of an MCDA framework such as Currently, the AIFA does consider some of the criteria EVIDEM could enable the collection of stakeholders’ included in the EVIDEM framework; however, this is not preferences (via weights allocation) and help to ensure that done systematically and it remains unclear how evidence they are taken into account more systematically in decision on those criteria is developed and to what extent it influ- making (via determination of the value score and its con- ences decision making. An MCDA process such as the one sideration in decision making). applied in this study can make both aspects more explicit The value score can help identify the key criteria and lead to more consistent consideration of multiple cri- impacting the intervention’s value, and lead to an in-depth teria in decision making. To implement MCDA in practice discussion within the decision-making committee around at national level, broader and larger groups of stakeholders the evidence presented on those criteria and the level of embracing different disease areas would need to be con- consensus that was obtained across participants when sulted. Alternatively, the decision-making committee (ei- assigning weights and scores. It can also inform sensitivity ther the Technical Scientific Committee or the Prices and analyses evaluating the robustness of the decision outcome. Reimbursement Committee within the AIFA) can act as the The value score has limited use in absolute terms if there agent and represent different stakeholders (such as the local is no specification on how it should inform coverage NHS payers, general public and patients—the principals) in decisions. If used to compare and rank competing tech- determining the relative importance of criteria. nologies, or across successive decisions about different technologies, score values might be useful. An example of this approach is the MCDA framework for health tech- 5 Limitations nology prioritisation developed for the Israeli Advisory Committee [23]. This approach used the benefit score and This study was exploratory as it applied MCDA in the the net total cost to draw efficiency frontiers and, based on context of medicine reimbursement decision making in budget constraints, selected the technologies to be funded. Italy in a convenience sample. In future uses, improve- On the other hand, for repeated reimbursement or HTA ments can be made to increase the validity of results and decisions affecting a fixed budget, there is a need to define their applicability in formal decision-making processes. the ‘hurdle for adoption’ [24]; in other words, the incre- Our convenience sample ensured a high rate of response mental cost per value score to compare against the cost per but its size and clinical areas covered could be expanded, value score of individual interventions to understand e.g. involving patients and clinicians of other non-oncology whether they are good value for money. However, given conditions. Survey instructions and synthesis of evidence the methodological issues in defining and estimating the on obinutuzumab were provided in English, while during opportunity cost of HTA decisions [25–28] and the role of the meetings, the participants and the moderator spoke in regional (as opposed to national) jurisdictions in the man- Italian, which helped in the interpretation of the scientific agement of the health budget, there might be a need to evidence and the instructions. If the exercise is conducted develop new approaches to ensure efficient decision mak- on a larger scale, it would benefit from the translation of all ing [22]. the material into the relevant language to increase under- In the context of the Italian National Health Service standing and rate of responses. (NHS), EVIDEM is implemented in the Lombardia region Patient representatives also raised the need to simplify to make listing and de-listing decisions on medical devices the language used to explain the framework and make it [7]. More than 20 interventions have been appraised and more accessible to lay persons. This shows that the inter- have obtained value scores, including economic criteria, active component is not only needed at the end of the between 0.22 and 0.72 [9]. Therefore, obinutuzumab fits in process, to consolidate survey responses, but also at the the middle of this range. However, the version of EVIDEM start, to ensure full understanding around the criteria and used in Lombardia is slightly different to that used in our their definitions. Challenges in communicating the elicita- study; it is applied in a regional context rather than a tion exercise to patients was also highlighted by Marsh national context, and information on which scores, on et al. [30]. Therefore, validation with stakeholders should average, allowed interventions to be approved for reim- be part of each step of an MCDA exercise [21]. bursement does not appear to be available. Given the importance of the interaction part of an We should also highlight that, consistent with the pur- MCDA process, future exercise should include qualitative poses of MCDA, value scores are intended to inform and analysis on methods used to structure group discussions support decision making and not to be used as a prescriptive and the type of interaction. Exploring how structured group rule in place of deliberations. A deliberative component is discussions change (or not) when participants meet in seen as necessary in all decision-making processes [29]. person compared with when they interact in a virtual space Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 161 (e.g. webinar) might help to establish efficient and effective domains and secondly among criteria within each domain, ways to support group dynamics. we noted some distortions for the values obtained by cri- In addition, as discussed in Mu ¨ hlbacher and Kaczynski teria in domains with two criteria, which tended to have [31], there are underlying difficulties in understanding the higher weights, versus criteria in domains with three cri- data provided and interpreting the score scales. For teria. One way to avoid this is to assign points directly to example, three patients reported that obinutuzumab was the criteria rather than splitting the task into two stages. For cheaper than the comparator (in the cost of intervention larger-scale applications of MCDA, alternative instruments criterion), which is not in line with what we would have to elicit preferences that have strong theoretical founda- expected from the data presented to them. This may reveal tions, and that have been used in other types of healthcare that they did not fully understand the task. In contrast, all assessments, can be considered, such as discrete choice patients reported that obinutuzumab was more clinically experiments and PAPRIKA [33]. effective than the comparator, which is in line with what we would have expected. The interval scale incorporated into EVIDEM has been 6 Conclusions tested and validated in several applications [31] and refined accordingly over time. For example, scores were initially To our knowledge, this is the first time that an MCDA measured on a scale from 0 to 3 [13] and subsequently on approach has been used to inform reimbursement decision scale from 0 to 5, probably to increase its discriminatory making of medicines at the national level in Italy. power. Others have suggested the use of a scale from 0 to This study showed that MCDA (in particular, EVIDEM) 100 [32]. Further validation exercises might be required to can be used to elicit the views of different stakeholder assess consistency and interval properties of the scoring groups. We found that the views of patients and clinicians scale. were broadly aligned as they expressed preference for We observe that more guidance needs to be provided in interventions targeting severe conditions and they ranked order to score some of the criteria, particularly the eco- economic criteria as the least important interventions. nomic-related criteria. What constitutes ‘substantial addi- Payers allocated higher weight (compared with patients tional expenditures’ (corresponding to a score of -5) for and clinicians) to the economic criteria and to the quality of one respondent might be different to that of another evidence. The key criteria driving the value of obinu- respondent. We included information about the national tuzumab accordingly to all the stakeholder groups were pharmaceutical expenditure to provide some context; disease severity, type of therapeutic benefit and unmet however, clear ranges or cut-off values should be included needs. for each score to ensure consistency in responses. Decision makers in Italy already consider some of the The validation process should also ensure that criteria EVIDEM criteria, such as disease severity, but with no are ‘preference independent’, meaning that it is possible to systematic approach. The perspectives of stakeholders judge how well one criterion is achieved without knowing (such as patients) are not elicited or incorporated at any how well any of the other criteria are achieved. One par- stage of the assessment or decision-making process. Our ticipant pointed out that the criterion ‘unmet needs’ should MCDA study provides useful evidence to decision makers, be considered in conjunction with ‘disease severity’ as the such as the AIFA, on what health interventions attribute lack of alternative interventions is meaningless to decision different stakeholders’ value the most, and has tested makers if it is not referred to a serious condition. This issue methods to ensure that this is captured consistently across can be addressed with alternative aggregation approaches, different interventions. such as multiplicative instead of additive methods [21]. To limit the cognitive burden of participants, we focused the meeting discussions on a restricted number of criteria Author contributions MG and ND designed the study. MG con- (the 13 EVIDEM ‘core criteria’). There is no rule on the ducted the structured meetings, with input from NAM and AP. GH set up the survey, analysed the data in Microsoft Excel, and reviewed the optimal number of criteria to include in an MCDA evidence included in the product evidence synthesis report. All framework; however, it is important to consider the trade- authors contributed to the drafting of the manuscript. off between breath/inclusiveness of a framework and resources needed to develop relevant evidence and analyse Compliance with Ethical Standards it for decision making [21]. Funding The work on this paper was funded by Roche S.P.A. with On the method to obtain weights, we selected one of the the intention of publishing the results from the outset. methods recommended by the EVIDEM collaboration: the point allocation approach. However, because of the top- Conflict of interest Martina Garau, Grace Hampson and Nancy Devlin are employed by The Office of Health Economics, which down approach used to allocate points, firstly among the 162 M. Garau et al. receives funding from a variety of sources, including the Association context-specific adaptation. Pharmacoeconomics. of the British Pharmaceutical Industry. Nicola Amedeo Mazzanti and 2016;34(3):285–301. Antonio Profico are employees of Roche S.P.A. 12. Daniels N, Sabin JE. Accountability for reasonableness: an update. BMJ. 2008;337:a1850. 13. Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P, Data availability statement Our input data and the material circu- et al. Bridging health technology assessment (HTA) with multi- lated to the study participants is available as electronic supplementary criteria decision analyses (MCDA): field testing of the EVIDEM material. framework for coverage decisions by a public payer in Canada. BMC Health Serv Res. 2011;11:329. Open Access This article is distributed under the terms of the 14. Gilabert-Perramon A, Catalan A, Prat A, Fontanet M, Puig-Peiro ´ Creative Commons Attribution-NonCommercial 4.0 International R, Merino-Montero S, et al. Development of a multi-criteria License (http://creativecommons.org/licenses/by-nc/4.0/), which per- decision analysis (MCDA) framework for health care decision- mits any noncommercial use, distribution, and reproduction in any making in Catalonia (Spain): pilot study focused in orphan drugs. medium, provided you give appropriate credit to the original Vienna: ISPOR 19th Annual European Congress; 2016. author(s) and the source, provide a link to the Creative Commons 15. van Til J, Groothuis-Oudshoorn C, Lieferink M, Dolan J, Goet- license, and indicate if changes were made. ghebeur M. Does technique matter: a pilot study exploring weighting techniques for a multi-criteria decision support framework. Cost Eff Resour Alloc. 2014;12:22. 16. Sehn LH, Chua N, Mayer J, Dueck G, Trneny M, Bouabdallah K, et al. Obinutuzumab plus bendamustine versus bendamustine References monotherapy in patients with rituximab-refractory indolent non- Hodgkin lymphoma (GADOLIN): a randomised, controlled, 1. HTAi Patient and Citizen Involvement. Good Practice Examples open-label, multicentre, phase 3 trial. Lancet Oncol. of Patient and Public Involvement in Health Technology 2016;17(8):1081–93. Assessment. 2015. http://www.htai.org/fileadmin/HTAi_Files/ 17. European Medicines Agency. Assessment report Gazyvaro Procedure ISG/PatientInvolvement/EffectiveInvolvement/Good_Practice_ No. EMEA/H/C/002799/II/0007. 2016. http://www.ema.europa.eu/ Examples_Feb_2015.pdf. Accessed 14 Aug 2017. docs/en_GB/document_library/EPAR_-_Assessment_Report_-_Vari 2. SMC. PACE—patient and clinician engagement. Factsheet. 2015. ation/human/002799/WC500211313.pdf. Accessed 14 Aug 2017. https://www.scottishmedicines.org.uk/files/PACE/PACE_facts 18. Lee L, Wang L, Crump M. Identification of potential surrogate heet_0417.pdf. Accessed 14 Aug 2017. end points in randomized clinical trials of aggressive and indolent non-Hodgkin’s lymphoma: correlation of complete response, 3. Shah KK, Mestre-Ferrandiz J, Towse A, Smyth EN. A review of time-to-event and overall survival end points. Ann Oncol. health technology appraisals: case studies in oncology. Int J 2011;22(6):1392–403. Technol Assess Health Care. 2013;29(1):101–9. 19. Belton V, Pictet J. A framework for group decision using a 4. Devlin N, Sussex J. Incorporating multiple criteria in HTA: MCDA model: sharing, aggregating or comparing individual methods and processes. OHE Monograph. London: Office of information? J Decis Syst. 1997;6(3):283–303. Health Economics; 2011. 20. Claxton K. Three questions to ask when examining MCDA. 5. Danner M, Hummel JM, Volz F, van Manen JG, Wiegard B, Value Outcomes Spotlight. 2015;1:10–8. Dintsios CM, et al. Integrating patients’ views into health tech- 21. Marsh K, IJmerzan M, Thokala P, Baltussen R, Boysen M, Kalo nology assessment: analytic hierarchy process (AHP) as a method Z, et al. Multiple criteria decision analysis for health care deci- to elicit patient preferences. Int J Technol Assess Health Care. sion making—emerging good practices: report 2 of the ISPOR 2011;27(4):369–75. MCDA emerging good practices task force. Value Health. 6. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, 2016;19(2):125–37. et al. Multiple criteria decision analysis for health care decision making-an introduction: report 1 of the ISPOR MCDA emerging 22. Garau M, Devlin NJ. Using MCDA as a decision aid in health good practices task force. Value Health. 2016;19(1):1–13. technology appraisal for coverage decisions: opportunities, 7. Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali challenges and unresolved questions. In: Marsh K, Goetghebeur M. Implementation of EUnetHTA core Model(R) in Lombardia: M, Thokala P, Baltussen R, editors. Multi-criteria decision the VTS framework. Int J Technol Assess Health Care. analysis to support healthcare decisions. Cham: Springer Inter- 2014;30(1):105–12. national Publishing; 2017. p. 277–98. 8. Castro H, Tringali M, Cleemput I, Devriese S, Leoni O, Lettieri 23. Golan O, Hansen P. Which health technologies should be funded? E. Advancing MCDA and HTA into coverage decision-making. A prioritization framework based explicitly on value for money. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, editors. Isr J Health Policy Res. 2012;1(1):44. Multi-criteria decision analysis to support healthcare decisions. 24. Sussex J, Towse A, Devlin N. Operationalizing value-based Cham: Springer International Publishing; 2017. p. 119–46. pricing of medicines: a taxonomy of approaches. Pharmacoeco- 9. Wahlster P, Goetghebeur M, Schaller S, Kriza C, Kolominsky- nomics. 2013;31(1):1–10. Rabas P. Exploring the perspectives and preferences for HTA 25. Culyer A, McCabe C, Briggs A, Claxton K, Buxton M, Akehurst across German healthcare stakeholders using a multi-criteria R, et al. Searching for a threshold, not setting one: the role of the assessment of a pulmonary heart sensor as a case study. Health National Institute for Health and Clinical Excellence. J Health Serv Res Policy. 2007;12(1):56–8. Res Policy Syst. 2015;13:24. 26. Karlsberg Schaffer S, Sussex J, Devlin N, Walker A. Local health 10. Bentley TG, Cohen JT, Elkin EB, Huynh J, Mukherjea A, Neville care expenditure plans and their opportunity costs. Health Policy. TH, et al. Validity and reliability of value assessment frameworks 2015;119(9):1237–44. for new cancer drugs. Value Health. 2017;20(2):200–5. 27. Appleby J, Devlin N, Parkin D, Buxton M, Chalkidou K. 11. Wagner M, Khoury H, Willet J, Rindress D, Goetghebeur M. Can Searching for cost effectiveness thresholds in the NHS. Health the EVIDEM framework tackle issues raised by evaluating Policy. 2009;91(3):239–45. treatments for rare diseases: analysis of issues and policies, and Applying a MCDA Approach to Elicit Stakeholders’ Preferences… 163 28. Claxton K, Martin S, Soares M, Rice N, Spackman E, Hinde S, research and future development of MCDA. Appl Health Econ et al. Methods for the estimation of the National Institute for Health Policy. 2016;14(1):29–40. Health and Care Excellence cost-effectiveness threshold. Health 32. Phillips LD. Best Practice for MCDA in Healthcare. In: Marsh K, Technol Assess. 2015;19(14):1–503 (v–vi). Goetghebeur M, Thokala P, Baltussen R, editors. Multi-criteria 29. Baltussen R. Question is not whether but how to use MCDA. decision analysis to support healthcare decisions. Cham: Springer Value Outcomes Spotlight. 2015;1(1):14–6. International Publishing; 2017. p. 311–29. 30. Marsh K, Caro JJ, Hamed A, Zaiser E. Amplifying each patient’s 33. Golan O, Hansen P, Kaplan G, Tal O. Health technology prior- voice: a systematic review of multi-criteria decision analyses itization: which criteria for prioritizing new technologies and involving patients. Appl Health Econ Health Policy. what are their relative weights? Health Policy. 2017;15(2):155–62. 2011;102(2–3):126–35. 31. Mu ¨ hlbacher AC, Kaczynski A. Making good decisions in healthcare with multi-criteria decision analysis: the use, current

Journal

PharmacoEconomics - OpenSpringer Journals

Published: Aug 22, 2017

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off