ORIGINAL REPORT
MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment
Tommi Tervonen,
Heather Gelhorn, Sumitra Sri Bhashyam, Jiat-Ling Poon, Katharine S. Gries,
Anne Rentz, Kevin Marsh,
Corresponding Author
Tommi Tervonen
Evidera, London, UK
Correspondence
T. Tervonen, Evidera, London, UK.
Email: [email protected]
Search for more papers by this authorKatharine S. Gries
Janssen Global Services, Titusville, NJ, USA
Search for more papers by this authorTommi Tervonen,
Heather Gelhorn, Sumitra Sri Bhashyam, Jiat-Ling Poon, Katharine S. Gries,
Anne Rentz, Kevin Marsh,
Corresponding Author
Tommi Tervonen
Evidera, London, UK
Correspondence
T. Tervonen, Evidera, London, UK.
Email: [email protected]
Search for more papers by this authorKatharine S. Gries
Janssen Global Services, Titusville, NJ, USA
Search for more papers by this authorAbstract
Purpose
Multiple criteria decision analysis swing weighting (SW) and discrete choice experiments (DCE) are appropriate methods for capturing patient preferences on treatment benefit-risk trade-offs. This paper presents a qualitative comparison of the 2 methods.
Methods
We review and critically assess similarities and differences of SW and DCE based on 6 aspects: comprehension by study participants, cognitive biases, sample representativeness, ability to capture heterogeneity in preferences, reliability and validity, and robustness of the results.
Results
The SW choice task can be more difficult, but the workshop context in which SW is conducted may provide more support to patients who are unfamiliar with the end points being evaluated or who have cognitive impairments. Both methods are similarly prone to a number of biases associated with preference elicitation, and DCE is prone to simplifying heuristics, which limits its application with large number of attributes. The low cost per patient of the DCE means that it can be better at achieving a representative sample, though SW does not require such large sample sizes due to exact nature of the collected preference data. This also means that internal validity is automatically enforced with SW, while the internal validity of DCE results needs to be assessed manually.
Conclusions
Choice between the 2 methods depends on characteristics of the benefit-risk assessment, especially on how difficult the trade-offs are for the patients to make and how many patients are available. Although there exist some empirical studies on many of the evaluation aspects, critical evidence gaps remain.
REFERENCES
- 1Egbrink MO, IJzerman M. The value of quantitative patient preferences in regulatory benefit-risk assessment. J Mark Access Health Policy [Internet]. 2014 Apr 1;2. Available from: https://doi.org/10.3402/jmahp.v2.22761
- 2Eichler H-G, Abadie E, Baker M, Rasi G. Fifty years after thalidomide; what role for drug regulators? Br J Clin Pharmaco. 2012; 74(5): 731-733.
- 3Postmus D, Douwe P, Maria M, et al. Incorporating patient preferences into drug development and regulatory decision making: results from a quantitative pilot study with cancer patients, carers, and regulators. Clin Pharmacol Ther. 2016; 99(5): 548-554.
- 4 FDA. Patient preference information—voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling. Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders [Internet]. 2016 Aug [cited 2016 Sep 6]. Available from: http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM446680.pdf
- 5Guitouni A, Adel G, Jean-Marc M. Tentative guidelines to help choosing an appropriate MCDA method. Eur J Oper Res. 1998; 109(2): 501-521.
- 6Marsh K, IJzerman M, Thokala P, et al. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016a; 19(2): 125-137.
- 7Potoglou D, Dimitris P, Peter B, et al. Best–worst scaling vs. discrete choice experiments: an empirical comparison using social care data. Soc Sci Med. 2011; 72(10): 1717-1727.
- 8van Dijk JD, Groothuis-Oudshoorn CGM, Marshall DA, IJzerman MJ. An empirical comparison of discrete choice experiment and best-worst scaling to estimate stakeholders' risk tolerance for hip replacement surgery. Value Health. 2016; 19(4): 316-322.
- 9Severin F, Schmidtke J, Mühlbacher A, Rogowski WH. Eliciting preferences for priority setting in genetic testing: a pilot study comparing best-worst scaling and discrete-choice experiments. Eur J Hum Genet. 2013 Nov; 21(11): 1202-1208.
- 10Whitty JA, Ratcliffe J, Chen G, Scuffham PA. Australian public preferences for the funding of new health technologies: a comparison of discrete choice and profile case best-worst scaling methods. Med Decis Making. 2014 Jul; 34(5): 638-654.
- 11van Til J, Catharina G-O, Marijke L, James D, Mireille G. Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework. Cost Eff Resour Alloc. 2014; 12(1): 22
- 12van Til JA, Dolan JG, Stiggelbout AM, Groothuis KCGM, Ijzerman MJ. The use of multi-criteria decision analysis weight elicitation techniques in patients with mild cognitive impairment: a pilot study. Patient. 2008 Apr 1; 1(2): 127-135.
- 13Guo JJ, Pandey S, Doyle J, Bian B, Lis Y, Raisch DW. A review of quantitative risk-benefit methodologies for assessing drug safety and efficacy-report of the ISPOR risk-benefit management working group. Value Health. 2010 Aug; 13(5): 657-666.
- 14Hughes D, Waddingham E, Mt-Isa S, et al. Recommendations for benefit-risk assessment methodologies and visual representations. Pharmacoepidemiol Drug Saf. 2016; 25(3): 251-262.
- 15Reed Johnson F, Lancsar E, Marshall D, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR conjoint analysis experimental design good research practices task Force. Value Health. 2013 Jan; 16(1): 3-13.
- 16Hauber AB, González JM, Groothuis-Oudshoorn CGM, et al. Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force. Value Health. 2016 Jun; 19(4): 300-315.
- 17Bridges JFP, Hauber AB, Marshall D, et al. Conjoint analysis applications in health--a checklist: A report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011 Jun; 14(4): 403-413.
- 18Flynn TN, Louviere JJ, Peters TJ, Coast J. Best–worst scaling: what it can do for health care research and how to do it. J Health Econ. 2007 Jan; 26(1): 171-189.
- 19 Medical device innovation consortium (MDIC). Patient centered benefit-risk project report [Internet]. Available from: http://mdic.org/wp-content/uploads/2015/05/MDIC_PCBR_Framework_Web.pdf
- 20De Montis A, De Toro P, Droste-Franke B, Stagl S. Assessing the quality of different MCDA methods. In: M Getzner, CL Spash, S Stagl, eds. Alternatives for Environmental Valuation. Routledge; 2005. p: 99-133.
- 21Von Neumann J, Morgenstern O. Theory of Games and Economic Behavior. Princeton, NJ: Princeton University Press; 1953: 641 p.
- 22Amaya-Amaya M, Gerard K, Ryan M. Discrete choice experiments in a nutshell. In: M Ryan, K Gerard, M Amaya-Amaya, eds. Using Discrete Choice Experiments to Value Health and Health Care. Springer; 2008. p. 13-46.
- 23Louviere JJ, Flynn TN, Carson RT. Discrete choice experiments are not conjoint analysis. Journal of Choice Modelling. 2010; 3(3): 57-72.
- 24Manski CF. The structure of random utility models. Theory Decis. 1977; 8(3): 229-254.
- 25McFadden D. Econometric models of probabilistic choice. In: CF Manski, ed. And Structural Analysis of Discrete Data with Econometric Applications. MIT Press; 1981. p: 198-272.
- 26Goodwin P, Wright G. Decision Analysis for Management Judgment. Chicester, UK: John Wiley & Sons; 2007: 492 p.
- 27Pöyhönen M, Hämäläinen RP. On the convergence of multiattribute weighting methods. Eur J Oper Res. 2001; 129(3): 569-585.
- 28Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S. Helping doctors and patients make sense of health statistics. Psychol Sci Public Interest. 2007 Nov; 8(2): 53-96.
- 29Hibbard JH, Peters E. Supporting informed consumer health care decisions: Data presentation approaches that facilitate the use of information in choice. Annu Rev Public Health. 2003; 24: 413-433.
- 30Peters E, Vastfjall D, Slovic P, Mertz CK, Mazzocco K, Dickert S. Numeracy and decision making. Psychol Sci. 2006; 17(5): 407-413.
- 31Knapp P. Comparison of two methods of presenting risk information to patients about the side effects of medicines. Qual Saf Health Care. 2004; 13(3): 176-180.
- 32Peters E, Hibbard J, Slovic P, Dieckmann N. Numeracy skill and the communication, comprehension, and use of risk-benefit information. Health Aff. 2007a May; 26(3): 741-748.
- 33Gigerenzer G. Collective statistical illiteracy: a cross-cultural comparison with probabilistic national samples. Arch Intern Med. 2010; 170(5): 468
- 34Waters EA, Weinstein ND, Colditz GA, Emmons K. Formats for improving risk communication in medical tradeoff decisions. J Health Commun. 2006 Mar; 11(2): 167-182.
- 35Morton A, Fasolo B. Behavioural decision theory for multi-criteria decision analysis: A guided tour ☆. J Oper Res Soc. 2008; 60(2): 268-275.
- 36Montibeller G, von Winterfeldt D. Cognitive and motivational biases in decision and risk analysis. Risk Anal. 2015 Jul; 35(7): 1230-1251.
- 37Day B, Bateman IJ, Carson RT, et al. Ordering effects and choice set awareness in repeat-response stated preference studies. J Environ Econ Manage. 2012; 63: 73-91.
- 38Day B, Prades J-LP. Ordering anomalies in choice experiments. J Environ Econ Manage. 2010; 59(3): 271-285.
- 39Lahtinen TJ, Hämäläinen RP. Path dependence and biases in the even swaps decision analysis method. Eur J Oper Res. 2016; 249(3): 890-898.
- 40Veldwijk J, Jorien V, Domino D, et al. Exploring how individuals complete the choice tasks in a discrete choice experiment: an interview study. BMC Med Res Methodol [Internet]. 2016; 16(1). https://doi.org/10.1186/s12874-016-0140-4
- 41Ozdemir S. Improving the validity of stated-preference data in health research: the potential of the time-to-think approach. Patient. 2015 Jun; 8(3): 247-255.
- 42Peters E, Dieckmann N, Dixon A, Hibbard JH, Mertz CK. Less is more in presenting quality information to consumers. Med Care Res Rev. 2007b Apr; 64(2): 169-190.
- 43Marshall D, Bridges JFP, Hauber B, et al. Conjoint analysis applications in health—how are studies being designed and reported?: an update on current practice in the published literature between 2005 and 2008. Patient. 2010 Dec 1; 3(4): 249-256.
- 44Bech M, Kjaer T, Lauridsen J. Does the number of choice sets matter? Results from a web survey applying a discrete choice experiment. Health Econ. 2011 Mar; 20(3): 273-286.
- 45Luce MF, Payne JW, Bettman JR. Emotional trade-off difficulty and choice. J Market Res. 1999; 36(2): 143
- 46Kadziński M, Greco S, Słowiński R. Selection of a representative value function for robust ordinal regression in group decision making. Group Decis Negot. 2011; 22(3): 429-462.
- 47Matsatsinis NF, Samaras AP. MCDA and preference disaggregation in group decision support systems. Eur J Oper Res. 2001; 130(2): 414-429.
- 48Carson RT, Groves T. Incentive and information properties of preference questions: commentary and extensions. In: J Bennett, ed. The International Handbook on Non-Market Environmental Valuation. Northampton, MA, USA: Edward Elgar; 2011: 300-321.
10.4337/9780857931191.00020 Google Scholar
- 49de Bekker-Grob EW, Donkers B, Jonker MF, Stolk EA. Sample size requirements for discrete-choice experiments in healthcare: a practical guide. Patient. 2015; 8(5): 373-384.
- 50Ho MP, Gonzalez JM, Lerner HP, et al. Incorporating patient-preference evidence into regulatory decision making. Surg Endosc. 2015 Oct; 29(10): 2984-2993.
- 51Flynn TN, Louviere JJ, Peters TJ, Coast J. Using discrete choice experiments to understand preferences for quality of life. Variance-scale heterogeneity matters. Soc Sci Med. 2010; 70(12): 1957-1965.
- 52Fiebig DG, Keane MP, Jordan L, Nada W. The generalized multinomial logit model: accounting for scale and coefficient heterogeneity. Market Sci. 2010; 29(3): 393-421.
- 53Kaltoft MK, Turner R, Cunich M, Salkeld G, Nielsen JB, Dowie J. Addressing preference heterogeneity in public health policy by combining cluster analysis and multi-criteria decision analysis: proof of method. Health Econ Rev. 2015; 5: 10.
- 54Kaltoft MK, Dowie J, Turner R, Nielsen JB, Salkeld G, Cunich M. Addressing the disconnect between public health science and personalised health care: the potential role of cluster analysis in combination with multi-criteria decision analysis. Lancet. 2013; 382: S52
- 55Kadziński M, Corrente S, Greco S, Słowiński R. Preferential reducts and constructs in robust multiple criteria ranking and sorting. OR Spectrum. 2014; 36(4): 1021-1053.
- 56Ryan M, Amaya-Amaya M. Concluding thoughts. In: M Ryan, K Gerard, M Amaya-Amaya, eds. Using Discrete Choice Experiments to Value Health and Health Care. Springer; 2008.
- 57Bijlenga D, Birnie E, Bonsel GJ. Feasibility, reliability, and validity of three health-state valuation methods using multiple-outcome vignettes on moderate-risk pregnancy at term. Value Health. 2009 Jul; 12(5): 821-827.
- 58Bryan S, Gold L, Sheldon R, Buxton M. Preference measurement using conjoint methods: an empirical investigation of reliability. Health Econ. 2000 Jul; 9(5): 385-395.
- 59Skjoldborg US, Lauridsen J, Junker P. Reliability of the discrete choice experiment at the input and output level in patients with rheumatoid arthritis. Value Health. 2009 Jan; 12(1): 153-158.
- 60Ryan M, Netten A, Skåtun D, Smith P. Using discrete choice experiments to estimate a preference-based measure of outcome—an application to social care for older people. J Health Econ. 2006 Sep; 25(5): 927-944.
- 61Clark MD, Determann D, Petrou S, Moro D, de Bekker-Grob EW. Discrete choice experiments in health economics: a review of the literature. Pharmacoeconomics. 2014 Sep; 32(9): 883-902.
- 62Lancsar E, Louviere J. Deleting “irrational” responses from discrete choice experiments: a case of investigating or imposing preferences? Health Econ. 2006 Aug; 15(8): 797-811.
- 63Lambooij MS, Harmsen IA, Veldwijk J, et al. Consistency between stated and revealed preferences: a discrete choice experiment and a behavioural experiment on vaccination behaviour compared. BMC Med Res Methodol. 2015 Mar 12; 15: 19
- 64Krucien N, Gafni A, Pelletier-Fleury N. Empirical testing of the external validity of a discrete choice experiment to determine preferred treatment option: the case of sleep apnea. Health Econ. 2015 Aug; 24(8): 951-965.
- 65Salampessy BH, Veldwijk J, Jantine Schuit A, et al. The predictive value of discrete choice experiments in public health: an exploratory application. Patient. 2015 Dec; 8(6): 521-529.
- 66Train KE. Discrete Choice Methods with Simulation 2nd ed. Cambridge University Press; 2003.
10.1017/CBO9780511753930 Google Scholar
- 67Figueira J, Greco S, Ehrgott M. Multiple criteria decision analysis: state of the art surveys. 2005.
- 68Lahdelma R, Salminen P. SMAA-2: stochastic multicriteria acceptability analysis for group decision making. Oper Res. 2001; 49(3): 444-454.
- 69Tervonen T, Figueira JR. A survey on stochastic multicriteria acceptability analysis methods. Journal of Multi-Criteria Decision Analysis. 2008; 15(1–2): 1-14.
10.1002/mcda.407 Google Scholar
- 70Tervonen T, van Valkenhoef G, Buskens E, Hillege HL, Postmus D. A stochastic multicriteria model for evidence-based decision making in drug benefit-risk analysis. Stat Med. 2011; 30(12): 1419-1428.
- 71Tervonen T, Naci H, van Valkenhoef G, et al. Applying multiple criteria decision analysis to comparative benefit-risk assessment: choosing among statins in primary prevention. Med Decis Making. 2015; 35(7): 859-871.
- 72van Valkenhoef G, Tervonen T, Zhao J, de Brock B, Hillege HL, Postmus D. Multicriteria benefit–risk assessment using network meta-analysis. J Clin Epidemiol. 2012; 65(4): 394-403.
- 73Greco S, Mousseau V, Słowiński R. Ordinal regression revisited: multiple criteria ranking using a set of additive value functions. Eur J Oper Res. 2008; 191(2): 416-436.
- 74Kadziński M, Greco S, Słowiński R. Extreme ranking analysis in robust ordinal regression. Omega. 2012; 40(4): 488-501.
- 75Corrente S, Greco S, Kadziński M, Słowiński R. Robust ordinal regression in preference learning and ranking. Mach Learn. 2013; 93(2–3): 381-422.
- 76Greco S, Kadziński M, Mousseau V, Słowiński R. Robust ordinal regression for multiple criteria group decision: UTA GMS-GROUP and UTADIS GMS-GROUP. Decis Support Syst. 2012; 52(3): 549-561.
- 77Kadziński M, Tervonen T. Stochastic ordinal regression for multiple criteria sorting problems. Decis Support Syst. 2013a; 55(1): 55-66.
- 78Kadziński M, Tervonen T. Robust multi-criteria ranking with additive value models and holistic pair-wise preference statements. Eur J Oper Res. 2013b; 228(1): 169-180.
- 79Keeney RL, Raiffa H. Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Cambridge University Press; 1993: 569 p.
10.1017/CBO9781139174084 Google Scholar
- 80Hensher DA, Rose JM, Greene WH. Applied Choice Analysis. Cambridge, UK: Cambridge University Press; 2005.
10.1017/CBO9780511610356 Google Scholar
Citing Literature
