Considering bias and conflicts of interest among the included studies
Isabelle Boutron
Search for more papers by this authorMatthew J Page
Search for more papers by this authorJulian PT Higgins
Search for more papers by this authorDouglas G Altman
Search for more papers by this authorAndreas Lundh
Search for more papers by this authorAsbjørn Hróbjartsson
Search for more papers by this authoron behalf of the Cochrane Bias Methods Group
Search for more papers by this authorIsabelle Boutron
Search for more papers by this authorMatthew J Page
Search for more papers by this authorJulian PT Higgins
Search for more papers by this authorDouglas G Altman
Search for more papers by this authorAndreas Lundh
Search for more papers by this authorAsbjørn Hróbjartsson
Search for more papers by this authoron behalf of the Cochrane Bias Methods Group
Search for more papers by this authorJulian P.T. Higgins
Search for more papers by this authorJames Thomas
Search for more papers by this authorJacqueline Chandler
Search for more papers by this authorMiranda Cumpston
Search for more papers by this authorTianjing Li
Search for more papers by this authorMatthew J. Page
Search for more papers by this authorVivian A. Welch
Search for more papers by this authorSummary
Bias can arise because of the actions of primary study investigators or because of the actions of review authors, or may be unavoidable due to constraints on how research can be undertaken in practice. Where possible, assessments of risk of bias in a systematic review should be informed by evidence. This chapter summarizes some of the key evidence about bias that informs our guidance on risk-of-bias assessments in Cochrane Reviews. Many results are often available in trial reports, so review authors should think carefully about which results to assess for risk of bias. Review authors who restrict their primary analysis in this way are encouraged to perform sensitivity analyses to show how conclusions might be affected if studies at a high risk of bias were included. The chapter also addresses how source of funding and conflicts of interest of study authors may impact on study design, conduct and reporting.
References
- Ahn R , Woodbridge A , Abraham A , Saba S , Korenstein D , Madden E , Boscardin WJ , Keyhani S . Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study . BMJ 2017 ; 356 : i6770 .
- Als-Nielsen B , Chen W , Gluud C , Kjaergard LL . Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? JAMA 2003 ; 290 : 921 – 928 .
- Bero LA , Grundy Q . Why having a (nonfinancial) interest is not a conflict of interest . PLoS Biology 2016 ; 14 : e2001221 .
- Blümle A , Meerpohl JJ , Schumacher M , von Elm E . Fate of clinical research studies after ethical approval – follow-up of study protocols until publication . PloS One 2014 ; 9 : e87184 .
- Boutron I , Dutton S , Ravaud P , Altman DG . Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes . JAMA 2010 ; 303 : 2058 – 2064 .
- Chan A-W , Song F , Vickers A , Jefferson T , Dickersin K , Gøtzsche PC , Krumholz HM , Ghersi D , van der Worp HB . Increasing value and reducing waste: addressing inaccessible research . The Lancet 2014 ; 383 : 257 – 266 .
- Chan AW , Hróbjartsson A , Haahr MT , Gøtzsche PC , Altman DG . Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles . JAMA 2004a ; 291 : 2457 – 2465 .
- Chan AW , Krleža-Jeric K , Schmid I , Altman DG . Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research . Canadian Medical Association Journal 2004b ; 171 : 735 – 740 .
- da Costa BR , Beckett B , Diaz A , Resta NM , Johnston BC , Egger M , Jüni P , Armijo-Olivo S . Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study . Systematic Reviews 2017 ; 6 : 44 .
- Dechartres A , Boutron I , Trinquart L , Charles P , Ravaud P . Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study . Annals of Internal Medicine 2011 ; 155 : 39 – 51 .
- Dechartres A , Trinquart L , Boutron I , Ravaud P . Influence of trial sample size on treatment effect estimates: meta-epidemiological study . BMJ 2013 ; 346 : f2304 .
- Dechartres A , Trinquart L , Faber T , Ravaud P . Empirical evaluation of which trial characteristics are associated with treatment effect estimates . Journal of Clinical Epidemiology 2016a ; 77 : 24 – 37 .
- Dechartres A , Ravaud P , Atal I , Riveros C , Boutron I . Association between trial registration and treatment effect estimates: a meta-epidemiological study . BMC Medicine 2016b ; 14 : 100 .
- Dechartres A , Trinquart L , Atal I , Moher D , Dickersin K , Boutron I , Perrodeau E , Altman DG , Ravaud P . Evolution of poor reporting and inadequate methods over time in 20,920 randomised controlled trials included in Cochrane reviews: research on research study . BMJ 2017 ; 357 : j2490 .
- Dechartres A , Atal I , Riveros C , Meerpohl J , Ravaud P . Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study . Annals of Internal Medicine 2018 ; 169 : 385 – 393 .
- Drazen JM , de Leeuw PW , Laine C , Mulrow C , DeAngelis CD , Frizelle FA , Godlee F , Haug C , Hébert PC , Horton R , Kotzin S , Marusic A , Reyes H , Rosenberg J , Sahni P , Van der Weyden MB , Zhaori G . Towards more uniform conflict disclosures: the updated ICMJE conflict of interest reporting form . BMJ 2010 ; 340 : c3239 .
- Duyx B , Urlings MJE , Swaen GMH , Bouter LM , Zeegers MP . Scientific citations favor positive results: a systematic review and meta-analysis . Journal of Clinical Epidemiology 2017 ; 88 : 92 – 101 .
- Easterbrook PJ , Berlin JA , Gopalan R , Matthews DR . Publication bias in clinical research . The Lancet 1991 ; 337 : 867 – 872 .
- Estellat C , Ravaud P . Lack of head-to-head trials and fair control arms: randomized controlled trials of biologic treatment for rheumatoid arthritis . Archives of Internal Medicine 2012 ; 172 : 237 – 244 .
- Eyding D , Lelgemann M , Grouven U , Harter M , Kromp M , Kaiser T , Kerekes MF , Gerken M , Wieseler B . Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials . BMJ 2010 ; 341 : c4737 .
- Fanelli D , Costas R , Ioannidis JPA . Meta-assessment of bias in science . Proceedings of the National Academy of Sciences of the United States of America 2017 ; 114 : 3714 – 3719 .
- Franco A , Malhotra N , Simonovits G . Social science. Publication bias in the social sciences: unlocking the file drawer . Science 2014 ; 345 : 1502 – 1505 .
- Gates A , Vandermeer B , Hartling L . Technology-assisted risk of bias assessment in systematic reviews: a prospective cross-sectional evaluation of the RobotReviewer machine learning tool . Journal of Clinical Epidemiology 2018 ; 96 : 54 – 62 .
- Grundy Q , Dunn AG , Bourgeois FT , Coiera E , Bero L . Prevalence of disclosed conflicts of interest in biomedical research and associations with journal impact factors and altmetric scores . JAMA 2018 ; 319 : 408 – 409 .
- Guyatt GH , Oxman AD , Vist GE , Kunz R , Falck-Ytter Y , Alonso-Coello P , Schünemann HJ . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations . BMJ 2008 ; 336 : 924 – 926 .
- Hakoum MB , Jouni N , Abou-Jaoude EA , Hasbani DJ , Abou-Jaoude EA , Lopes LC , Khaldieh M , Hammoud MZ , Al-Gibbawi M , Anouti S , Guyatt G , Akl EA . Characteristics of funding of clinical trials: cross-sectional survey and proposed guidance . BMJ Open 2017 ; 7 : e015997 .
- Hartling L , Hamm MP , Milne A , Vandermeer B , Santaguida PL , Ansari M , Tsertsvadze A , Hempel S , Shekelle P , Dryden DM . Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs . Journal of Clinical Epidemiology 2013 ; 66 : 973 – 981 .
- Higgins JPT , Altman DG , Gøtzsche PC , Jüni P , Moher D , Oxman AD , Savović J , Schulz KF , Weeks L , Sterne JAC . The Cochrane Collaboration's tool for assessing risk of bias in randomised trials . BMJ 2011 ; 343 : d5928 .
-
Hopewell S
,
Clarke M
,
Stewart L
,
Tierney J
.
Time to publication for results of clinical trials
.
Cochrane Database of Systematic Reviews
2007
;
2
: MR000011.
10.1002/14651858.MR000006.pub2 Google Scholar
- Hopewell S , Boutron I , Altman D , Ravaud P . Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study . BMJ Open 2013 ; 3 : 8 .
- Jefferson T , Jones MA , Doshi P , Del Mar CB , Hama R , Thompson MJ , Onakpoya I , Heneghan CJ . Risk of bias in industry-funded oseltamivir trials: comparison of core reports versus full clinical study reports . BMJ Open 2014 ; 4 : e005253 .
- Jones CW , Keil LG , Holland WC , Caughey MC , Platts-Mills TF . Comparison of registered and published outcomes in randomized controlled trials: a systematic review . BMC Medicine 2015 ; 13 : 282 .
- Jørgensen L , Paludan-Muller AS , Laursen DR , Savović J , Boutron I , Sterne JAC , Higgins JPT , Hróbjartsson A . Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews . Systematic Reviews 2016 ; 5 : 80 .
- Jüni P , Witschi A , Bloch R , Egger M . The hazards of scoring the quality of clinical trials for meta-analysis . JAMA 1999 ; 282 : 1054 – 1060 .
- Jüni P , Altman DG , Egger M . Systematic reviews in health care: assessing the quality of controlled clinical trials . BMJ 2001 ; 323 : 42 – 46 .
- Kirkham JJ , Dwan KM , Altman DG , Gamble C , Dodd S , Smyth R , Williamson PR . The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews . BMJ 2010 ; 340 : c365 .
- Li G , Abbade LPF , Nwosu I , Jin Y , Leenus A , Maaz M , Wang M , Bhatt M , Zielinski L , Sanger N , Bantoto B , Luo C , Shams I , Shahid H , Chang Y , Sun G , Mbuagbaw L , Samaan Z , Levine MAH , Adachi JD , Thabane L . A systematic review of comparisons between protocols or registrations and full reports in primary biomedical research . BMC Medical Research Methodology 2018 ; 18 : 9 .
- Lo B , Field MJ , Institute of Medicine (US) Committee on Conflict of Interest in Medical Research Education and Practice . Conflict of Interest in Medical Research, Education, and Practice . Washington, D.C .: National Academies Press (US) ; 2009 .
- Lundh A , Lexchin J , Mintzes B , Schroll JB , Bero L . Industry sponsorship and research outcome . Cochrane Database of Systematic Reviews 2017 ; 2 : MR000033.
- Mann H , Djulbegovic B . Comparator bias: why comparisons must address genuine uncertainties . Journal of the Royal Society of Medicine 2013 ; 106 : 30 – 33 .
- Marret E , Elia N , Dahl JB , McQuay HJ , Møiniche S , Moore RA , Straube S , Tramèr MR . Susceptibility to fraud in systematic reviews: lessons from the Reuben case . Anesthesiology 2009 ; 111 : 1279 – 1289 .
- Marshall IJ , Kuiper J , Wallace BC . RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials . Journal of the American Medical Informatics Association 2016 ; 23 : 193 – 201 .
- McGauran N , Wieseler B , Kreis J , Schuler YB , Kolsch H , Kaiser T . Reporting bias in medical research – a narrative review . Trials 2010 ; 11 : 37 .
- Millard LA , Flach PA , Higgins JPT . Machine learning to assist risk-of-bias assessments in systematic reviews . International Journal of Epidemiology 2016 ; 45 : 266 – 277 .
- Moher D , Shamseer L , Cobey KD , Lalu MM , Galipeau J , Avey MT , Ahmadzai N , Alabousi M , Barbeau P , Beck A , Daniel R , Frank R , Ghannad M , Hamel C , Hersi M , Hutton B , Isupov I , McGrath TA , McInnes MDF , Page MJ , Pratt M , Pussegoda K , Shea B , Srivastava A , Stevens A , Thavorn K , van Katwyk S , Ward R , Wolfe D , Yazdi F , Yu AM , Ziai H . Stop this waste of people, animals and money . Nature 2017 ; 549 : 23 – 25 .
- Montedori A , Bonacini MI , Casazza G , Luchetta ML , Duca P , Cozzolino F , Abraha I . Modified versus standard intention-to-treat reporting: are there differences in methodological quality, sponsorship, and findings in randomized trials? A cross-sectional study. Trials 2011 ; 12 : 58 .
- Morgan AJ , Ross A , Reavley NJ . Systematic review and meta-analysis of Mental Health First Aid training: effects on knowledge, stigma, and helping behaviour . PloS One 2018 ; 13 : e0197102 .
- Morrison A , Polisena J , Husereau D , Moulton K , Clark M , Fiander M , Mierzwinski-Urban M , Clifford T , Hutton B , Rabb D . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies . International Journal of Technology Assessment in Health Care 2012 ; 28 : 138 – 144 .
- Norris SL , Burda BU , Holmer HK , Ogden LA , Fu R , Bero L , Schunemann H , Deyo R . Author's specialty and conflicts of interest contribute to conflicting guidelines for screening mammography . Journal of Clinical Epidemiology 2012 ; 65 : 725 – 733 .
- Odutayo A , Emdin CA , Hsiao AJ , Shakir M , Copsey B , Dutton S , Chiocchia V , Schlussel M , Dutton P , Roberts C , Altman DG , Hopewell S . Association between trial registration and positive study findings: cross-sectional study (Epidemiological Study of Randomized Trials-ESORT) . BMJ 2017 ; 356 : j917 .
- Page MJ , Higgins JPT . Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study . Systematic Reviews 2016 ; 5 : 108 .
- Page MJ , Higgins JPT , Clayton G , Sterne JAC , Hróbjartsson A , Savović J . Empirical evidence of study design biases in randomized trials: systematic review of meta-epidemiological studies . PloS One 2016 ; 11 : 7 .
- Page MJ , McKenzie JE , Higgins JPT . Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review . BMJ Open 2018 ; 8 : e019703 .
- Patel SV , Yu D , Elsolh B , Goldacre BM , Nash GM . Assessment of conflicts of interest in robotic surgical studies: validating author's declarations with the open payments database . Annals of Surgery 2018 ; 268 : 86 – 92 .
- Polanin JR , Tanner-Smith EE , Hennessy EA . Estimating the difference between published and unpublished effect sizes: a meta-review . Review of Educational Research 2016 ; 86 : 207 – 236 .
- Rasmussen K , Schroll J , Gøtzsche PC , Lundh A . Under-reporting of conflicts of interest among trialists: a cross-sectional study . Journal of the Royal Society of Medicine 2015 ; 108 : 101 – 107 .
- Riechelmann RP , Wang L , O'Carroll A , Krzyzanowska MK . Disclosure of conflicts of interest by authors of clinical trials and editorials in oncology . Journal of Clinical Oncology 2007 ; 25 : 4642 – 4647 .
- Rising K , Bacchetti P , Bero L . Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation . PLoS Medicine 2008 ; 5 : e217 .
- Riveros C , Dechartres A , Perrodeau E , Haneef R , Boutron I , Ravaud P . Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals . PLoS Medicine 2013 ; 10 : e1001566 .
- Rothenstein JM , Tomlinson G , Tannock IF , Detsky AS . Company stock prices before and after public announcements related to oncology drugs . Journal of the National Cancer Institute 2011 ; 103 : 1507 – 1512 .
- Safer DJ . Design and reporting modifications in industry-sponsored comparative psychopharmacology trials . Journal of Nervous and Mental Disease 2002 ; 190 : 583 – 592 .
- Saini P , Loke YK , Gamble C , Altman DG , Williamson PR , Kirkham JJ . Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews . BMJ 2014 ; 349 : g6501 .
- Sampson M , Barrowman NJ , Moher D , Klassen TP , Pham B , Platt R , St John PD , Viola R , Raina P . Should meta-analysts search Embase in addition to Medline? Journal of Clinical Epidemiology 2003 ; 56 : 943 – 955 .
- Savović J , Jones HE , Altman DG , Harris RJ , Jüni P , Pildal J , Als-Nielsen B , Balk EM , Gluud C , Gluud LL , Ioannidis JPA , Schulz KF , Beynon R , Welton NJ , Wood L , Moher D , Deeks JJ , Sterne JAC . Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials . Annals of Internal Medicine 2012 ; 157 : 429 – 438 .
- Scherer RW , Meerpohl JJ , Pfeifer N , Schmucker C , Schwarzer G , von Elm E . Full publication of results initially presented in abstracts . Cochrane Database of Systematic Reviews 2018 ; 11 : MR000005.
- Schmid CH . Outcome reporting bias: a pervasive problem in published meta-analyses . American Journal of Kidney Diseases 2016 ; 69 : 172 – 174 .
- Schmucker C , Schell LK , Portalupi S , Oeller P , Cabrera L , Bassler D , Schwarzer G , Scherer RW , Antes G , von Elm E , Meerpohl JJ . Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries . PloS One 2014 ; 9 : e114023 .
- Schulz KF , Chalmers I , Hayes RJ , Altman DG . Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials . JAMA 1995 ; 273 : 408 – 412 .
- Shawwa K , Kallas R , Koujanian S , Agarwal A , Neumann I , Alexander P , Tikkinen KA , Guyatt G , Akl EA . Requirements of clinical journals for authors’ disclosure of financial and non-financial conflicts of interest: a cross-sectional study . PloS One 2016 ; 11 : e0152301 .
- Sterne JAC . Why the Cochrane risk of bias tool should not include funding source as a standard item [editorial] . Cochrane Database of Systematic Reviews 2013 ; 12 : ED000076.
- Tramèr MR , Reynolds DJ , Moore RA , McQuay HJ . Impact of covert duplicate publication on meta-analysis: a case study . BMJ 1997 ; 315 : 635 – 640 .
- Turner RM , Spiegelhalter DJ , Smith GC , Thompson SG . Bias modelling in evidence synthesis . Journal of the Royal Statistical Society Series A (Statistics in Society) 2009 ; 172 : 21 – 47 .
- Urrutia G , Ballesteros M , Djulbegovic B , Gich I , Roque M , Bonfill X . Cancer randomized trials showed that dissemination bias is still a problem to be solved . Journal of Clinical Epidemiology 2016 ; 77 : 84 – 90 .
- Vedula SS , Li T , Dickersin K . Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin . PLoS Medicine 2013 ; 10 : e1001378 .
- Viswanathan M , Carey TS , Belinson SE , Berliner E , Chang SM , Graham E , Guise JM , Ip S , Maglione MA , McCrory DC , McPheeters M , Newberry SJ , Sista P , White CM . A proposed approach may help systematic reviews retain needed expertise while minimizing bias from nonfinancial conflicts of interest . Journal of Clinical Epidemiology 2014 ; 67 : 1229 – 1238 .
- Welton NJ , Ades AE , Carlin JB , Altman DG , Sterne JAC . Models for potentially biased evidence in meta-analysis using empirically based priors . Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009 ; 172 : 119 – 136 .
- Wieland LS , Berman BM , Altman DG , Barth J , Bouter LM , D'Adamo CR , Linde K , Moher D , Mullins CD , Treweek S , Tunis S , van der Windt DA , Zwarenstein M , Witt C . Rating of included trials on the efficacy-effectiveness spectrum: development of a new tool for systematic reviews . Journal of Clinical Epidemiology 2017 ; 84 .
- Wieseler B , Kerekes MF , Vervoelgyi V , McGauran N , Kaiser T . Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications . BMJ 2012 ; 344 : d8141 .
- Wood L , Egger M , Gluud LL , Schulz K , Jüni P , Altman DG , Gluud C , Martin RM , Wood AJG , Sterne JAC . Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study . BMJ 2008 ; 336 : 601 – 605 .
- Zarin DA , Tse T , Williams RJ , Carr S . Trial reporting in ClinicalTrials.gov – The Final Rule . New England Journal of Medicine 2016 ; 375 : 1998 – 2004 .