Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ
Corresponding Author
Danielle L. Burke
Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
Correspondence to: Danielle L Burke, Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
E-mail: [email protected]
Search for more papers by this authorJoie Ensor
Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
Search for more papers by this authorRichard D. Riley
Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
Search for more papers by this authorCorresponding Author
Danielle L. Burke
Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
Correspondence to: Danielle L Burke, Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
E-mail: [email protected]
Search for more papers by this authorJoie Ensor
Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
Search for more papers by this authorRichard D. Riley
Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, U.K.
Search for more papers by this authorAbstract
Meta-analysis using individual participant data (IPD) obtains and synthesises the raw, participant-level data from a set of relevant studies. The IPD approach is becoming an increasingly popular tool as an alternative to traditional aggregate data meta-analysis, especially as it avoids reliance on published results and provides an opportunity to investigate individual-level interactions, such as treatment-effect modifiers. There are two statistical approaches for conducting an IPD meta-analysis: one-stage and two-stage. The one-stage approach analyses the IPD from all studies simultaneously, for example, in a hierarchical regression model with random effects. The two-stage approach derives aggregate data (such as effect estimates) in each study separately and then combines these in a traditional meta-analysis model. There have been numerous comparisons of the one-stage and two-stage approaches via theoretical consideration, simulation and empirical examples, yet there remains confusion regarding when each approach should be adopted, and indeed why they may differ.
In this tutorial paper, we outline the key statistical methods for one-stage and two-stage IPD meta-analyses, and provide 10 key reasons why they may produce different summary results. We explain that most differences arise because of different modelling assumptions, rather than the choice of one-stage or two-stage itself. We illustrate the concepts with recently published IPD meta-analyses, summarise key statistical software and provide recommendations for future IPD meta-analyses. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.
References
- 1Debray TP, Moons KG, van Valkenhoef G, Efthimiou O, Hummel N, Groenwold RH, Reitsma JB, GetReal Methods Review Group. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Research Synthesis Methods 2015; 6(4): 293–309.
- 2Stewart GB, Altman DG, Askie LM, Duley L, Simmonds MC, Stewart LA. Statistical analysis of individual participant data meta-analyses: a comparison of methods and recommendations for practice. PloS One 2012; 7(10): e46042.
- 3Debray TP, Moons KG, Abo-Zaid GM, Koffijberg H, Riley RD. Individual participant data meta-analysis for a binary outcome: one-stage or two-stage? PloS One 2013; 8(4): e60650.
- 4Berlin JA, Santanna J, Schmid CH, Szczech LA, Feldman HI, Anti-Lymphocyte Antibody Induction Therapy Study Group. Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Statistics in Medicine 2002; 21(3): 371–387.
- 5Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ 2010; 340: c221.
- 6Simmonds M, Stewart G, Stewart L. A decade of individual participant data meta-analyses: a review of current practice. Contemporary Clinical Trials 2015; 45(Pt A): 76–83.
- 7Simmonds MC, Higgins JP, Stewart LA, Tierney JF, Clarke MJ, Thompson SG. Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clinical Trials 2005; 2(3): 209–217.
- 8Abo-Zaid G, Guo B, Deeks JJ, Debray TP, Steyerberg EW, Moons KG, Riley RD. Individual participant data meta-analyses should not ignore clustering. Journal of Clinical Epidemiology 2013; 66(8): 865–873.
- 9Bowden J, Tierney JF, Simmonds M, Copas AJ, Higgins JPT. Individual patient data meta-analysis of time-to-event outcomes: one-stage versus two-stage approaches for estimating the hazard ratio under a random effects model. Research Synthesis Methods 2011; 2(3): 150–162.
- 10Higgins JPT, Green S ( ). Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration, 2011. Available from: www.cochrane-handbook.org.
- 11Hamza TH, van Houwelingen HC, Stijnen T. The binomial distribution of meta-analysis was preferred to model within-study variability. Journal of Clinical Epidemiology 2008; 61(1): 41–51.
- 12Thompson S, Kaptoge S, White I, Wood A, Perry P, Danesh J, Emerging Risk Factors Collaboration. Statistical methods for the time-to-event analysis of individual participant data from multiple epidemiological studies. International Journal of Epidemiology 2010; 39(5): 1345–1359.
- 13Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M, Rovers M. Individual participant data (IPD) meta-analyses of randomised controlled trials: guidance on their use. PLoS Medicine 2015; 12(7): e1001855.
- 14Bland M. An Introduction to Medical Statistics ( 4th edn). Oxford University Press: New York, 2015.
- 15Zou G. A modified Poisson regression approach to prospective studies with binary data. American Journal of Epidemiology 2004; 159(7): 702–706.
- 16Whitehead A, Whitehead J. A general parametric approach to the meta-analysis of randomized clinical trials. Statistics in Medicine 1991; 10(11): 1665–1677.
- 17DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials 1986; 7(3): 177–188.
- 18 The Cochrane Collaboration. Review Manager (RevMan). The Nordic Cochrane Centre: Copenhagen, 5.3. 2014.
- 19Kontopantelis E, Reeves D. Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: a simulation study. Statistical Methods in Medical Research 2012; 21(4): 409–426.
- 20DerSimonian R, Kacker R. Random-effects model for meta-analysis of clinical trials: an update. Contemporary Clinical Trials 2007; 28(2): 105–114.
- 21Langan D, Higgins JP, Simmonds M. An empirical comparison of heterogeneity variance estimators in 12 894 meta-analyses. Research Synthesis Methods 2015; 6(2): 195–205.
- 22IntHout J, Ioannidis JP, Borm GF. The Hartung–Knapp–Sidik–Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian–Laird method. BMC Medical Research Methodology 2014; 14: 25.
- 23Veroniki AA, Jackson D, Viechtbauer W, Bender R, Bowden J, Knapp G, Kuss O, Higgins JPT, Langan D, Salanti G. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Research Synthesis Methods 2016; 7(1): 55–79.
- 24Schmid CH, Stark PC, Berlin JA, Landais P, Lau J. Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. Journal of Clinical Epidemiology 2004; 57(7): 683–697.
- 25Senn S. The many modes of meta. Drug Information Journal 2000; 34: 535–549.
- 26Crowther MJ, Riley RD, Staessen JA, Wang J, Gueyffier F, Lambert PC. Individual patient data meta-analysis of survival data using Poisson regression models. BMC Medical Research Methodology 2012; 12: 34.
- 27Tudur Smith C, Williamson PR, Marson AG. An overview of methods and empirical comparison of aggregate data and individual patient data results for investigating heterogeneity in meta-analysis of time-to-event outcomes. Journal of Evaluation in Clinical Practice 2005; 11(5): 468–478.
- 28Sargent DJ. A general framework for random effects survival analysis in the Cox proportional hazards setting. Biometrics 1998; 54(4): 1486–1497.
- 29Tudur Smith C, Williamson PR, Marson AG. Investigating heterogeneity in an individual patient data meta-analysis of time to event outcomes. Statistics in Medicine 2005; 24(9): 1307–1319.
- 30Michiels S, Baujat B, Mahé C, Sargent DJ, Pignon JP. Random effects survival models gave a better understanding of heterogeneity in individual patient data meta-analyses. Journal of Clinical Epidemiology 2005; 58(3): 238–245.
- 31Debray TP, Moons KG, Ahmed I, Koffijberg H, Riley RD. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Statistics in Medicine 2013; 32(18): 3158–3180.
- 32Royston P, Parmar MK, Sylvester R. Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Statistics in Medicine 2004; 23(6): 907–926.
- 33Crowther MJ, Look MP, Riley RD. Multilevel mixed effects parametric survival models using adaptive Gauss–Hermite quadrature with application to recurrent events and individual participant data meta-analysis. Statistics in Medicine 2014; 33(22): 3844–3858.
- 34 SAS Institute Inc., Base SAS® 9.4 Procedures Guide. 2011, SAS Institute Inc. Cary, NC.
- 35 StataCorp, Stata Statistical Software: Release 14. 2015, StataCorp LP. College Station, TX.
- 36 R Core Team. r: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing: Vienna, Austria, 2013.
- 37Rasbash J, Charlton C, Browne WJ, Healy M, Cameron B. MLwiN. 2.1, 2009, Centre for Multilevel Modelling. University of Bristol.
- 38Kontopantelis E, Reeves D. A short guide and a forest plot command (ipdforest) for one-stage meta-analysis. The Stata Journal 2013; 13(3): 574–587.
- 39Fisher DJ. Two-stage individual participant data meta-analysis and generalized forest plots. The Stata Journal 2015; 15(2): 369–396.
- 40Harris RJ, Bradburn MJ, Deeks JJ, Harbord RM, Altman DG, Sterne JAC. metan: Fixed- and random-effects meta-analysis. The Stata Journal 2008; 8(1): 3–28.
- 41Viechtbauer W. Conducting meta-analyses in r with the metafor package. Journal of Statistical Software 2010; 36(3): 1–48.
- 42Kontopantelis E, Reeves D. metaan: Random-effects meta-analysis. The Stata Journal 2010; 10(3): 395–407.
- 43Harbord RM, Higgins JP. Meta-regression in Stata. The Stata Journal 2008; 8(4): 493–519.
- 44White IR. Multivariate random-effects meta-analysis. The Stata Journal 2009; 9(1): 40–56.
- 45 TM Palmer, JAC Sterne (Eds). Meta-Analysis in Stata: An Updated Collection from the Stata Journal ( Second edn). Stata Press: College Station, Texas, 2016.
- 46Littell RC, Milliken GA, Stroup WW, Wolfinger RD, Schabenberger O. SAS® for Mixed Models ( Second edn). SAS Institute Inc.: Cary, NC, USA, 2006.
- 47van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Statistics in Medicine 2002; 21(4): 589–624.
- 48Koopman L, van der Heijden GJ, Hoes AW, Grobbee DE, Rovers MM. Empirical comparison of subgroup effects in conventional and individual patient data meta-analyses. International Journal of Technology Assessment in Health Care 2008; 24(3): 358–361.
- 49Tudur Smith C, Williamson PR. A comparison of methods for fixed effects meta-analysis of individual patient data with time to event outcomes. Clinical Trials 2007; 4(6): 621–630.
- 50Steinberg KK, Smith SJ, Stroup DF, Olkin I, Lee NC, Williamson GD, Thacker SB. Comparison of effect estimates from a meta-analysis of summary data from published studies and from a meta-analysis using individual patient data for ovarian cancer studies. American Journal of Epidemiology 1997; 145(10): 917–925.
- 51Senn S. Hans van Houwelingen and the art of summing up. Biometrical Journal 2010; 52(1): 85–94.
- 52Mathew T, Nordstrom K. On the equivalence of meta-analysis using literature and using individual patient data. Biometrics 1999; 55(4): 1221–1223.
- 53Mathew T, Nordstrom K. Comparison of one-step and two-step meta-analysis models using individual patient data. Biometrical Journal 2010; 52(2): 271–287.
- 54Olkin I, Sampson A. Comparison of meta-analysis versus analysis of variance of individual patient data. Biometrics 1998; 54(1): 317–322.
- 55Riley RP, MJ, Jackson D, Wardle M, Gueyffier F, Wang J, Staessen JA, White IR. Multivariate meta-analysis using individual participant data. Research Synthesis Methods 2015; 6: 157–174.
- 56Stijnen T, Hamza TH, Ozdemir P. Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Statistics in Medicine 2010; 29(29): 3046–3067.
- 57Turner RM, Omar RZ, Yang M, Goldstein H, Thompson SG. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Statistics in Medicine 2000; 19(24): 3417–3432.
10.1002/1097-0258(20001230)19:24<3417::AID-SIM614>3.0.CO;2-L CAS PubMed Web of Science® Google Scholar
- 58Burke DL, Billingham LJ, Girling AJ, Riley RD. Meta-analysis of randomized phase II trials to inform subsequent phase III decisions. Trials 2014; 15: 346.
- 59Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23: 1351–1375.
- 60Bradburn MJ, Deeks JJ, Berlin JA, Russell Localio A. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Statistics in Medicine 2007; 26(1): 53–77.
- 61Rücker G, Schwarzer G, Carpenter J, Olkin I. Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Statistics in Medicine 2009; 28(5): 721–738.
- 62Takwoingi Y, Guo B, Riley RD, Deeks JJ. Performance of methods for meta-analysis of diagnostic test accuracy with few studies or sparse data. Statistical Methods in Medical Research 2015. doi:10.1177/0962280215592269.
- 63Simel DL, Bossuyt PM. Differences between univariate and bivariate models for summarizing diagnostic accuracy may not be large. Journal of Clinical Epidemiology 2009; 62(12): 1292–1300.
- 64Riley RD, Ahmed I, Debray TPA, Willis BH, Noordzij JP, Higgins JPT, Deeks JJ. Summarising and validating test accuracy results across multiple studies for use in clinical practice. Statistics in Medicine 2015; 34(13): 2081–2103.
- 65Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Progress in Cardiovascular Diseases 1985; 27(5): 335–371.
- 66Greenland S, Salvan A. Bias in the one-step method for pooling study results. Statistics in Medicine 1990; 9(3): 247–252.
- 67Abo-Zaid G, Sauerbrei W, Riley RD. Individual participant data meta-analysis of prognostic factor studies: state of the art? BMC Medical Research Methodology 2012; 12: 56.
- 68Altman DG, Deeks JJ. Meta-analysis, Simpson's paradox, and the number needed to treat. BMC Medical Research Methodology 2002; 2: 3.
- 69Sidik K, Jonkman JN. A comparison of heterogeneity variance estimators in combining results of studies. Statistics in Medicine 2007; 26(9): 1964–1981.
- 70Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Statistics in Medicine 1999; 18(20): 2693–708.
10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V CAS PubMed Web of Science® Google Scholar
- 71Berkey CS, Hoaglin DC, Mosteller F, Colditz GA. A random-effects regression model for meta-analysis. Statistics in Medicine 1995; 14(4): 395–411.
- 72Austin PC. Estimating multilevel logistic regression models when the number of clusters is low: a comparison of different statistical software procedures. International Journal of Biostatistics 2010; 6: 1 Article 16. doi:10.2202/1557-4679.1195.
10.2202/1557-4679.1195 Google Scholar
- 73Maengseok NL, Y. REML estimation for binary data in GLMMs. Journal of Multivariate Analysis 2007; 98: 896–915.
- 74Brostrom GH, H. Generalized linear models with clustered data: fixed and random-effects models. Computational Statistics & Data Analysis 2011; 55: 3123–3134.
- 75Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. British Medical Journal 2011; 342: 964–967.
- 76Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine 2003; 22(17): 2693–2710.
- 77Hartung J. An alternative method for meta-analysis. Biometrical Journal 1999; 41: 901–916.
- 78Hartung J, Knapp G. On tests of the overall treatment effect in meta-analysis with normally distributed responses. Statistics in Medicine 2001; 20(12): 1771–1782.
- 79Hartung J, Knapp G. A refined method for the meta-analysis of controlled clinical trials with binary outcome. Statistics in Medicine 2001; 20(24): 3875–3889.
- 80Sidik K, Jonkman JN. A simple confidence interval for meta-analysis. Statistics in Medicine 2002; 21(21): 3153–3159.
- 81Sidik KJ, JN. On constructing confidence intervals for a standardized mean difference in meta-analysis. Communications in Statistics - Simulation and Computation 2003; 32: 1191–1203.
- 82Kenward MG, Roger JH. Small sample inference for fixed effects from restricted maximum likelihood. Biometrics 1997; 53(3): 983–997.
- 83Jackson D, Riley R, White IR. Multivariate meta-analysis: potential and promise. Statistics in Medicine 2011; 30(20): 2481–2498.
- 84Jones AP, Riley RD, Williamson PR, Whitehead A. Meta-analysis of individual patient data versus aggregate data from longitudinal clinical trials. Clinical Trials 2009; 6(1): 16–27.
- 85Riley RD. Multivariate meta-analysis: the effect of ignoring within-study correlation. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172(4): 789–811.
- 86Wilcock GK, Birks J, Whitehead A, Evans SJ. The effect of selegiline in the treatment of people with Alzheimer's disease: a meta-analysis of published trials. International Journal of Geriatric Psychiatry 2002; 17(2): 175–183.
- 87Jackson D, White IR, Price M, Copas J, Riley RD. Borrowing of strength and study weights in multivariate and network meta-analysis. Statistical Methods in Medical Research 2015. doi:10.1177/0962280215611702.
- 88Fisher DJ, Copas AJ, Tierney JF, Parmar MK. A critical review of methods for the assessment of patient-level interactions in individual participant data meta-analysis of randomized trials, and guidance for practitioners. Journal of Clinical Epidemiology 2011; 64(9): 949–967.
- 89Riley RD, Lambert PC, Staessen JA, Wang J, Gueyffier F, Thijs L, Boutitie F. Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Statistics in Medicine 2008; 27(11): 1870–1893.
- 90Simmonds M. Statistical Methodology of Individual Patient Data Meta-analysis. University of Cambridge, 2005.
- 91Riley RD, Steyerberg EW. Meta-analysis of a binary outcome using individual participant data and aggregate data. Research Synthesis Methods 2010; 1(1): 2–19.
- 92Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009; 172(Part 1): 137–159.
- 93Koopman L, van der Heijden GJ, Grobbee DE, Rovers MM. Comparison of methods of handling missing data in individual patient data meta-analyses: an empirical example on antibiotics in children with acute otitis media. American Journal of Epidemiology 2008; 167(5): 540–545.
- 94Burgess S, White IR, Resche-Rigon M, Wood AM. Combining multiple imputation and meta-analysis with individual participant data. Statistics in Medicine 2013; 32(26): 4499–4514.
- 95Jolani S, Debray TP, Koffijberg H, van Buuren S, Moons KG. Imputation of systematically missing predictors in an individual participant data meta-analysis: a generalized approach using MICE. Statistics in Medicine 2015; 34(11): 1841–1863.
- 96White IR, Welton NJ, Wood AM, Ades AE, Higgins JP. Allowing for uncertainty due to missing data in meta-analysis – part 2: hierarchical models. Statistics in Medicine 2008; 27(5): 728–745.
- 97Resche-Rigon M, White IR, Bartlett JW, Peters SA, Thompson SG, PROG-IMT Study Group. Multiple imputation for handling systematically missing confounders in meta-analysis of individual participant data. Statistics in Medicine 2013; 32(28): 4890–4905.
- 98Quartagno M, Carpenter JR. Multiple imputation for IPD meta-analysis: allowing for heterogeneity and studies with missing covariates. Statistics in Medicine 2016; 35(17): 2938–2954.
- 99Sauerbrei W, Royston P. A new strategy for meta-analysis of continuous covariates in observational studies. Statistics in Medicine 2011; 30(28): 3341–3360.
- 100Abo-Zaid, G., Individual patient data meta-analysis of prognostic factor studies. PhD Thesis. 2011, University of Birmingham.