Extending inferences from a randomized trial to a new target population
Corresponding Author
Issa J. Dahabreh
Center for Evidence Synthesis in Health, Brown University, Providence, Rhode Island
Department of Health Services, Policy & Practice, Brown University, Providence, Rhode Island
Department of Epidemiology, Brown University, Providence, Rhode Island
Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
Issa J. Dahabreh MD ScD, Box G-S121-8; Brown University, Providence, RI 02912.
Email: [email protected]
Search for more papers by this authorSarah E. Robertson
Center for Evidence Synthesis in Health, Brown University, Providence, Rhode Island
Department of Health Services, Policy & Practice, Brown University, Providence, Rhode Island
Search for more papers by this authorJon A. Steingrimsson
Department of Biostatistics, School of Public Health, Brown University, Providence, Rhode Island
Search for more papers by this authorElizabeth A. Stuart
Departments of Mental Health, Biostatistics, and Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
Search for more papers by this authorMiguel A. Hernán
Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
Harvard-MIT Division of Health Sciences and Technology, Boston, Massachusetts
Search for more papers by this authorCorresponding Author
Issa J. Dahabreh
Center for Evidence Synthesis in Health, Brown University, Providence, Rhode Island
Department of Health Services, Policy & Practice, Brown University, Providence, Rhode Island
Department of Epidemiology, Brown University, Providence, Rhode Island
Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
Issa J. Dahabreh MD ScD, Box G-S121-8; Brown University, Providence, RI 02912.
Email: [email protected]
Search for more papers by this authorSarah E. Robertson
Center for Evidence Synthesis in Health, Brown University, Providence, Rhode Island
Department of Health Services, Policy & Practice, Brown University, Providence, Rhode Island
Search for more papers by this authorJon A. Steingrimsson
Department of Biostatistics, School of Public Health, Brown University, Providence, Rhode Island
Search for more papers by this authorElizabeth A. Stuart
Departments of Mental Health, Biostatistics, and Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
Search for more papers by this authorMiguel A. Hernán
Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
Harvard-MIT Division of Health Sciences and Technology, Boston, Massachusetts
Search for more papers by this authorAbstract
When treatment effect modifiers influence the decision to participate in a randomized trial, the average treatment effect in the population represented by the randomized individuals will differ from the effect in other populations. In this tutorial, we consider methods for extending causal inferences about time-fixed treatments from a trial to a new target population of nonparticipants, using data from a completed randomized trial and baseline covariate data from a sample from the target population. We examine methods based on modeling the expectation of the outcome, the probability of participation, or both (doubly robust). We compare the methods in a simulation study and show how they can be implemented in software. We apply the methods to a randomized trial nested within a cohort of trial-eligible patients to compare coronary artery surgery plus medical therapy versus medical therapy alone for patients with chronic coronary artery disease. We conclude by discussing issues that arise when using the methods in applied analyses.
CONFLICT OF INTEREST
The authors have no conflicts of interest to report.
Supporting Information
Filename | Description |
---|---|
SIM_8426-Supp-0001-APPENDIX_transporting_to_a_new_target_population__Stat_Med_revision_-2.pdfPDF document, 396.1 KB | SIM_8426-Supp-0001-APPENDIX_transporting_to_a_new_target_population__Stat_Med_revision_-2.pdf |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- 1Rothwell PM. External validity of randomised controlled trials: to whom do the results of this trial apply? The Lancet. 2005; 365(9453): 82-93.
- 2Evans A, Kalra L. Are the results of randomized controlled trials on anticoagulation in patients with atrial fibrillation generalizable to clinical practice? Arch Intern Med. 2001; 161(11): 1443-1447.
- 3Elting LS, Cooksley C, Bekele BN, et al. Generalizability of cancer clinical trial results. Cancer. 2006; 106(11): 2452-2458.
- 4Steg PG, López-Sendón J, de Sa EL, et al. External validity of clinical trials in acute myocardial infarction. Arch Intern Med. 2007; 167(1): 68-73.
- 5Dans AL, Dans LF, Guyatt GH, Richardson S. Users' guides to the medical literature: XIV. How to decide on the applicability of clinical trial results to your patient. JAMA. 1998; 279(7): 545-549.
- 6Cole SR, Stuart EA. Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial. Am J Epidemiol. 2010; 172(1): 107-115.
- 7Kaizar EE. Estimating treatment effect via simple cross design synthesis. Statist Med. 2011; 30(25): 2986-3009.
- 8O'Muircheartaigh C, Hedges LV. Generalizing from unrepresentative experiments: a stratified propensity score approach. J R Stat Soc C Appl Stat. 2014; 63(2): 195-210.
- 9Tipton E. Improving generalizations from experiments using propensity score subclassification: assumptions, properties, and contexts. J Educ Behav Stat. 2012; 38(3): 239-266.
- 10Tipton E, Hedges L, Vaden-Kiernan M, Borman G, Sullivan K, Caverly S. Sample selection in randomized experiments: a new method using propensity score stratified sampling. J Res Educ Eff. 2014; 7(1): 114-135.
- 11Hartman E, Grieve R, Ramsahai R, Sekhon JS. From SATE to PATT: combining experimental with observational studies to estimate population treatment effects. J R Stat Soc A Stat Soc. 2013; 10: 1111.
- 12Zhang Z, Nie L, Soon G, Hu Z. New methods for treatment effect calibration, with applications to non-inferiority trials. Biometrics. 2016; 72(1): 20-29.
- 13Rudolph KE, van der Laan MJ. Robust estimation of encouragement design intervention effects transported across sites. J R Stat Soc B Stat Methodol. 2017; 79(5): 1509-1525.
10.1111/rssb.12213 Google Scholar
- 14Westreich D, Edwards JK, Lesko CR, Stuart E, Cole SR. Transportability of trial results using inverse odds of sampling weights. Am J Epidemiol. 2017; 186(8): 1010-1014.
- 15Buchanan AL, Hudgens MG, Cole SR, et al. Generalizing evidence from randomized trials using inverse probability of sampling weights. J R Stat Soc A Stat Soc. 2018; 181(4): 1193-1209.
- 16Dahabreh IJ, Robertson SE, Tchetgen Tchetgen EJ, Stuart EA, Hernán MA. Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals. Biometrics. 2019; 75(2): 685-694.
- 17Dahabreh IJ, Hernán MA. Extending inferences from a randomized trial to a target population. Eur J Epidemiol. 2019; 34(8): 719-722.
- 18Hernán M. Discussion of perils and potentials of self-selected entry to epidemiological studies and surveys. J R Stat Soc A Stat Soc. 2016; 179(2): 346-347.
- 19Dahabreh IJ, Haneuse SJP, Robins JM, et al. Study designs for extending causal inferences from a randomized trial to a target population. 2019. arXiv preprint arXiv:1905.07764.
- 20Olschewski M, Scheurlen H. Comprehensive cohort study: an alternative to randomized consent design in a breast preservation trial. Methods Inf Med. 1985; 24(3): 131-134.
- 21Fiore LD, Lavori PW. Integrating randomized comparative effectiveness research with patient care. N Engl J Med. 2016; 374(22): 2152-2158.
- 22Ray WA. Evaluating medication effects outside of clinical trials: new-user designs. Am J Epidemiol. 2003; 158(9): 915-920.
- 23Splawa-Neyman J. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. [Translated from Splawa-Neyman, J (1923) in Roczniki Nauk Rolniczych Tom X, 1–51]. Statistical Science. 1990; 5(4): 465-472.
10.1214/ss/1177012031 Google Scholar
- 24Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol. 1974; 66(5): 688-701.
- 25Hernán MA, Robins JM. Causal Inference. Boca Raton, FL: Chapman & Hall/CRC; 2020. Forthcoming.
- 26Dahabreh IJ, Hayward R, Kent DM. Using group data to treat individuals: understanding heterogeneous treatment effects in the age of precision medicine and patient-centred evidence. Int J Epidemiol. 2016; 45(6): 2184-2193.
- 27Dahabreh IJ, Trikalinos TA, Kent DM, Schmid CH. Heterogeneity of treatment effects. Methods Comp Eff Res. 2017: 227.
- 28Dahabreh IJ, Hernán MA, Robertson SE, Buchanan A, Steingrimsson JA. Generalizing trial findings in nested trial designs with sub-sampling of non-randomized individuals. 2019. arXiv preprint arXiv:1902.06080.
- 29Dahabreh IJ, Robins JM, Haneuse SJP, Hernán MA. Generalizing causal inferences from randomized trials: counterfactual and graphical identification. 2019. arXiv preprint arXiv:1906.10792.
- 30Petersen ML, Porter KE, Gruber S, Wang Y, van der Laan MJ. Diagnosing and responding to violations in the positivity assumption. Stat Methods Med Res. 2012; 21(1): 31-54.
- 31Robins JM. A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect. Mathematical Modelling. 1986; 7(9-12): 1393-1512.
- 32Horvitz DG, Thompson DJ. A generalization of sampling without replacement from a finite universe. J Am Stat Assoc. 1952; 47(260): 663-685.
- 33Hájek J. Comment on An essay on the logical foundations of survey sampling by D. Basu. In: VP Godambe, DA Sprott, eds. Foundations of Statistical Inference. New York City, NY: Holt, Rinehart, and Winston; 1971: 236.
- 34Robins JM, Sued M, Lei-Gomez Q, Rotnitzky A. Comment: performance of double-robust estimators when inverse probability weights are highly variable. Statistical Science. 2007; 22(4): 544-559.
10.1214/07-STS227D Google Scholar
- 35Robins JM, Rotnitzky A, Zhao LP. Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc. 1994; 89(427): 846-866.
- 36Robins JM, Rotnitzky A. Semiparametric efficiency in multivariate regression models with missing data. J Am Stat Assoc. 1995; 90(429): 122-129.
- 37Robins JM, Rotnitzky A. Comment. Statistica Sinica. 2001; 11(4): 920-936.
- 38Gourieroux C, Monfort A, Trognon A. Pseudo maximum likelihood methods: theory. Econom J Econom Soc. 1984; 52(3): 681-700.
- 39Wooldridge JM. Inverse probability weighted estimation for general missing data problems. J Econom. 2007; 141(2): 1281-1301.
- 40Tsiatis A. Semiparametric Theory and Missing Data. Berlin, Germany: Springer Science & Business Media; 2007.
- 41Kang JD, Schafer JL. Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science. 2007; 22(4): 523-539.
- 42Stefanski LA, Boos DD. The calculus of M-estimation. Am Stat. 2002; 56(1): 29-38.
- 43Lunceford JK, Davidian M. Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statist Med. 2004; 23(19): 2937-2960.
- 44Chen Z, Kaizar E. On variance estimation for generalizing from a trial to a target population. 2017. arXiv preprint arXiv:1704.07789.
- 45Efron B, Tibshirani RJ. An introduction to the bootstrap. In: Monographs on Statistics and Applied Probability. Vol 57. Boca Raton, FL: Chapman & Hall/CRC; 1994.
10.1201/9780429246593 Google Scholar
- 46Austin PC, Stafford J. The performance of two data-generation processes for data with specified marginal treatment odds ratios. Commun Stat Simul Comput. 2008; 37(6): 1039-1051.
- 47 R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2015. https://www.R-project.org
- 48Saul BC, Hudgens MG. The calculus of M-estimation in R with geex. 2017. arXiv preprint arXiv:1709.01413.
- 49Chaitman BR, Ryan TJ, Kronmal RA, Foster ED, Frommer PL, Killip T. Coronary artery surgery study (CASS): comparability of 10 year survival in randomized and randomizable patients. J Am Coll Cardiol. 1990; 16(5): 1071-1078.
- 50Olschewski M, Schumacher M, Davis KB. Analysis of randomized and nonrandomized patients in clinical trials using the comprehensive cohort follow-up study design. Control Clin Trials. 1992; 13(3): 226-239.
- 51William J, Russell R, Nicholas T, et al. Coronary artery surgery study (CASS): a randomized trial of coronary artery bypass surgery. Circulation. 1983; 68(5): 939-950.
- 52 CASS Principal Investigators and Their Associates. Coronary artery surgery study (CASS): a randomized trial of coronary artery bypass surgery: comparability of entry characteristics and survival in randomized patients and nonrandomized patients meeting randomization criteria. J Am Coll Cardiol. 1984; 3(1): 114-128.
- 53Harrell F Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. Berlin, Germany: Springer; 2015.
10.1007/978-3-319-19425-7 Google Scholar
- 54Nguyen TQ, Ebnesajjad C, Cole SR, Stuart EA. Sensitivity analysis for an unobserved moderator in RCT-to-target-population generalization of treatment effects. Ann Appl Stat. 2017; 11(1): 225-247.
- 55Nguyen TQ, Ackerman B, Schmid I, Cole SR, Stuart EA. Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: assumptions, models, effect scales, data scenarios, and implementation details. PLOS ONE. 2018; 13(12):e0208795.
- 56Dahabreh IJ, Robins JM, Haneuse SJ-PA, et al. Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population. 2019. arXiv preprint arXiv:1905.10684.
- 57Rubin DB. Using propensity scores to help design observational studies: application to the tobacco litigation. Health Serv Outcomes Res Methodol. 2001; 2(3-4): 169-188.
10.1023/A:1020363010465 Google Scholar
- 58Waernbaum I, Pazzagli L. Model misspecification and bias for inverse probability weighting and doubly robust estimators. 2017. arXiv preprint arXiv:1711.09388.
- 59Chernozhukov V, Chetverikov D, Demirer M, et al. Double/debiased machine learning for treatment and structural parameters. Econom J. 2018; 21(1): C1-C68.
- 60Robins JM, Ritov Y. Toward a curse of dimensionality appropriate (CODA) asymptotic theory for semi-parametric models. Statist Med. 1997; 16(3): 285-319.
10.1002/(SICI)1097-0258(19970215)16:3<285::AID-SIM535>3.0.CO;2-# CAS PubMed Web of Science® Google Scholar
- 61Cole SR, Hernán MA. Constructing inverse probability weights for marginal structural models. Am J Epidemiol. 2008; 168(6): 656-664.
- 62Brookhart MA, van der Laan MJ. A semiparametric model selection criterion with applications to the marginal structural model. Comput Stat Data Anal. 2006; 50(2): 475-498.
- 63Vansteelandt S, Bekaert M, Claeskens G. On model selection and model misspecification in causal inference. Stat Methods Med Res. 2012; 21(1): 7-30.
- 64Bareinboim E, Pearl J. Meta-transportability of causal effects: a formal approach. In: Proceedings of the 16th International Conference on Artificial Intelligence and Statistics (AISTATS); 2013; Scottsdale, AZ.
- 65Pearl J, Bareinboim E. External validity: from do-calculus to transportability across populations. Statistical Science. 2014; 29(4): 579-595.
- 66Bareinboim E, Pearl J. Transportability of causal effects: completeness results. Paper presented at: Twenty-Sixth AAAI Conference on Artificial Intelligence; 2012; Toronto, Canada.
- 67Rotnitzky A, Robins JM, Scharfstein DO. Semiparametric regression for repeated outcomes with nonignorable nonresponse. J Am Stat Assoc. 1998; 93(444): 1321-1339.
- 68Robins JM, Rotnitzky A, Scharfstein DO. Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In: Statistical Models in Epidemiology, the Environment, and Clinical Trials. Berlin, Germany: Springer; 2000: 1-94.
10.1007/978-1-4612-1284-3_1 Google Scholar
- 69Bang H, Robins JM. Doubly robust estimation in missing data and causal inference models. Biometrics. 2005; 61(4): 962-973.
- 70Cao W, Tsiatis AA, Davidian M. Improving efficiency and robustness of the doubly robust estimator for a population mean with incomplete data. Biometrika. 2009; 96(3): 723-734.
- 71Vermeulen K, Vansteelandt S. Bias-reduced doubly robust estimation. J Am Stat Assoc. 2015; 110(511): 1024-1036.