Scientific Review
Predictors of Increased Fragility Index Scores in Surgical Randomized Controlled Trials: An Umbrella Review
Prushoth Vivekanantha,
Ajay Shah,
Graeme Hoit,
Olufemi Ayeni,
Daniel Whelan,
Prushoth Vivekanantha
Michael DeGroote School of Medicine, McMaster University, Hamilton, ON, Canada
Search for more papers by this authorCorresponding Author
Ajay Shah
Division of Orthopaedic Surgery, Postgraduate Medical Education, University of Toronto, Toronto, ON, Canada
Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, 149 College St, M5T 1P5 Toronto, ON, Canada
[email protected]Search for more papers by this authorGraeme Hoit
Division of Orthopaedic Surgery, Postgraduate Medical Education, University of Toronto, Toronto, ON, Canada
Search for more papers by this authorOlufemi Ayeni
Division of Orthopaedic Surgery, Department of Surgery, McMaster University, Hamilton, ON, Canada
Search for more papers by this authorDaniel Whelan
Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, 149 College St, M5T 1P5 Toronto, ON, Canada
Search for more papers by this authorPrushoth Vivekanantha,
Ajay Shah,
Graeme Hoit,
Olufemi Ayeni,
Daniel Whelan,
Prushoth Vivekanantha
Michael DeGroote School of Medicine, McMaster University, Hamilton, ON, Canada
Search for more papers by this authorCorresponding Author
Ajay Shah
Division of Orthopaedic Surgery, Postgraduate Medical Education, University of Toronto, Toronto, ON, Canada
Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, 149 College St, M5T 1P5 Toronto, ON, Canada
[email protected]Search for more papers by this authorGraeme Hoit
Division of Orthopaedic Surgery, Postgraduate Medical Education, University of Toronto, Toronto, ON, Canada
Search for more papers by this authorOlufemi Ayeni
Division of Orthopaedic Surgery, Department of Surgery, McMaster University, Hamilton, ON, Canada
Search for more papers by this authorDaniel Whelan
Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, 149 College St, M5T 1P5 Toronto, ON, Canada
Search for more papers by this authorCopyright comment: Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Abstract
Background
The fragility index (FI) is defined as the minimum number of patients or subjects needed to switch experimental groups for statistical significance to be lost in a randomized control trial (RCT). This index is used to determine the robustness of a study's findings and recently as a measure of evaluating RCT quality. The objective of this review was to identify and describe published systematic reviews utilizing FI to evaluate surgical RCTs and to determine if there were common factors associated with higher FI values.
Methods
Three databases (PubMed, MEDLINE [Ovid], Embase) were searched, followed by a subsequent abstract/title and full-text screening to yield 50 reviews of surgical RCTs. Authors, year of publication, name of journal, study design, number of RCTs, subspecialty, sample size, median FI, patients lost to follow-up, and associations between variables and FI scores were collected.
Results
Among 1007 of 2214 RCTs in 50 reviews reporting FI (median sample size 100), the pooled median FI was 3 (IQR: 1–7). Most reviews investigated orthopaedic surgery RCTs (n = 32). There was a moderate correlation between FI and p value (r = 0.-413), a mild correlation between FI and sample size (r = 0.188), and a mild correlation between FI and event number (r = 0.129).
Conclusion
Based on a limited sample of systematic reviews, surgical RCT FI values are still low (2–5). Future RCTs in surgery require improvement to study design in order to increase the robustness of statistically significant findings.
References
- 1Maldonado DR, Go CC, Huang BH et al. The fragility index of hip arthroscopy randomized controlled trials: a systematic survey. Arthrosc Tech (2021) 37(6): 1983–198910.1016/j.arthro.2021.01.049
10.1016/j.arthro.2021.01.049 Google Scholar
- 2Khormaee S, Choe J, Ruzbarsky JJ et al. The fragility of statistically significant results in pediatric orthopaedic randomized controlled trials as quantified by the fragility index: a systematic review. J Pediatr Orthop (2018) 38(8): e418–e4232997933210.1097/BPO.0000000000001201
- 3Herndon CL, McCormick KL, Gazgalis A et al. Fragility Index as a measure of randomized clinical trial quality in adult reconstruction: a systematic review. Arthroplast Today (2021) 11: 239–25134692962851728610.1016/j.artd.2021.08.018
- 4Walsh M, Srinathan SK, McAuley DF et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a fragility Index. J Clin Epidemiol (2014) 67(6): 622–6282450814410.1016/j.jclinepi.2013.10.019
- 5Hegazy MA, El Nahas W, Roshdy S Surgical outcome of modified versus conventional parotidectomy in treatment of benign parotid tumors. J Surg Oncol (2011) 103(2): 163–1682125925110.1002/jso.21779
- 6Marasco D, Russo J, Izzo A et al. Static versus dynamic fixation of distal tibiofibular syndesmosis: a systematic review of overlapping meta-analyses. Knee Surg Sports Traumatol Arthrosc (2021) 29(11): 3534–35423445544810.1007/s00167-021-06721-6
- 7Landis JR, Koch GG The measurement of observer agreement for categorical data. Biometrics (1977) 33(1): 159–17484357110.2307/2529310
- 8Kung J, Chiappelli F, Cajulis OO et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J (2010) 4: 84–91210886862948145
- 9Anand S, Kainth D Fragility index of recently published meta-analyses in pediatric urology: a striking observation. Cureus (2021) 13(7343678258343562
- 10Bowers A, Meyer C, Tritz D et al. Assessing quality of randomized trials supporting guidelines for laparoscopic and endoscopic surgery. J Surg Res (2018) 224: 233–2392950684610.1016/j.jss.2017.11.061
- 11Budhiraja P, Kaplan B, Kalot M et al. Current state of evidence on kidney transplantation: how fragile are the results. Transplantation (2022) 106(2): 248–2563396602210.1097/TP.0000000000003805
- 12Carroll AH, Rigor P, Wright MA et al. Fragility of randomized controlled trials on treatment of proximal humeral fracture. J Shoulder Elbow Surg (2022) 31(8): 1610–16163524030210.1016/j.jse.2022.01.141
- 13Checketts JX, Scott JT, Meyer C et al. The robustness of trials that guide evidence-based orthopaedic surgery. J Bone Joint Surg Am (2018) 100(122991693810.2106/JBJS.17.01039
- 14Chertin L, Burman I, Haifler M Urologic oncology randomized controlled trials are frequently fragile—A review of the urology literature. Urol Oncol (2021) 39(10): 735.e1–735.e83410322510.1016/j.urolonc.2021.04.037
- 15Chin B, Copeland A, Gallo L et al. The fragility of statistically significant randomized controlled trials in plastic surgery. Plast Reconstr Surg (2019) 144(5): 1238–12453168877210.1097/PRS.0000000000006102
- 16Constant M, Trofa DP, Saltzman BM et al. The fragility of statistical significance in patellofemoral instability research: a systematic review. Am J Sports Med (2021) 10.1177/0363546521103920234633219
- 17Davey MS, Hurley ET, Doyle TR et al. The fragility index of statistically significant findings from randomized controlled trials comparing the management strategies of anterior shoulder instability. Am J Sports Med (2022) 10.1177/0363546522107726835414266
- 18Ehlers CB, Curley AJ, Fackler NP et al. The statistical fragility of hamstring versus patellar tendon autografts for anterior cruciate ligament reconstruction: a systematic review of comparative studies. Am J Sports Med (2021) 49(10): 2827–28333321155510.1177/0363546520969973
- 19Ehlers CB, Curley AJ, Fackler NP et al. The statistical fragility of single-bundle vs double-bundle autografts for ACL reconstruction: a systematic review of comparative studies. Orthop J Sports Med (2021) 9(12): 2325967121106462634988239872138910.1177/23259671211064626
- 20Ekhtiari S, Gazendam AM, Nucci NW et al. The fragility of statistically significant findings from randomized controlled trials in hip and knee arthroplasty. J Arthroplast (2021) 36(6): 2211–2218.e110.1016/j.arth.2020.12.015
- 21Evaniew N, Files C, Smith C et al. The fragility of statistically significant findings from randomized trials in spine surgery: a systematic survey. Spine J (2015) 15(10): 2188–21972607246410.1016/j.spinee.2015.06.004
- 22Forrester LA, Jang E, Lawson MM et al. Statistical fragility of surgical and procedural clinical trials in orthopaedic oncology. J Am Acad Orthop Surg Glob Res Rev (2020) 4(6): e19.00152326564787322779
- 23Forrester LA, McCormick KL, Bonsignore-Opp L et al. Statistical fragility of surgical clinical trials in orthopaedic trauma. J Am Acad Orthop Surg Glob Res Rev (2021) 5(11): e20.00197348078898608260
- 24Gnech M, Lovatt CA, McGrath M et al. Quality of reporting and fragility index for randomized controlled trials in the vesicoureteral reflux literature: Where do we stand?. J Pediatr Urol (2019) 15(3): 204–2123106096510.1016/j.jpurol.2019.02.014
- 25Go CC, Maldonado DR, Go BC et al. The fragility index of total hip arthroplasty randomized control trials: a systematic review. J Am Acad Orthop Surg (2022) 30(9): e741–e7503521341910.5435/JAAOS-D-21-00489
- 26Khan M, Evaniew N, Gichuru M et al. The fragility of statistically significant findings from randomized trials in sports surgery: a systematic survey. Am J Sports Med (2017) 45(9): 2164–21702789503810.1177/0363546516674469
- 27Kyriakides PW, Schultz BJ, Egol K et al. The fragility and reverse fragility indices of proximal humerus fracture randomized controlled trials: a systematic review. Eur J Trauma Emerg Surg (2021) 10.1007/s00068-021-01684-234056677
- 28Li B, Kong I, McGrath M et al. Evaluating the literature on preoperative androgen stimulation for hypospadias repair using the fragility index - can we trust observational studies. J Pediatr Urol (2021) 17(5): 661–6693451812210.1016/j.jpurol.2021.07.027
- 29McCormick KL, Tedesco LJ, Swindell HW et al. Statistical fragility of randomized clinical trials in shoulder arthroplasty. J Shoulder Elbow Surg (2021) 30(8): 1787–17933327132310.1016/j.jse.2020.10.028
- 30Megafu M, Megafu E The fragility of statistical findings in distal radius fractures: a systematic review of randomized controlled trials. Injury (2022) 53(10): 3352–33563585147410.1016/j.injury.2022.07.017
- 31Morris SC, Gowd AK, Agarwalla A et al. Fragility of statistically significant findings from randomized clinical trials of surgical treatment of humeral shaft fractures: a systematic review. World J Orthop (2022) 13(9): 825–83636189338951662210.5312/wjo.v13.i9.825
- 32Muthu S, Ramakrishnan E Fragility analysis of statistically significant outcomes of randomized control trials in spine surgery: a systematic review. Spine (Phila Pa 1976) (2021) 46(3): 198–2083275628510.1097/BRS.0000000000003645
- 33Narayan VM, Gandhi S, Chrouser K et al. The fragility of statistically significant findings from randomized controlled trials in the urological literature. BJU Int (2018) 122(1): 160–1662956939010.1111/bju.14210
- 34Nelms DW, Vargas HD, Bedi RS et al. When the p value doesn't cut it: the fragility index applied to randomized controlled trials in colorectal surgery. Dis Colon Rectum (2022) 65(2): 276–2833499042610.1097/DCR.0000000000002146
- 35Noel CW, McMullen C, Yao C et al. The fragility of statistically significant findings from randomized trials in head and neck surgery. Laryngoscope (2018) 128(9): 2094–21002968349410.1002/lary.27183
- 36Parisien RL, Dashe J, Cronin PK et al. Statistical significance in trauma research: too unstable to trust. J Orthop Trauma (2019) 33(12): e466–e4703135644310.1097/BOT.0000000000001595
- 37Parisien RL, Trofa DP, Dashe J et al. Statistical fragility and the role of p values in the sports medicine literature. J Am Acad Orthop Surg (2019) 27(7): e324–e3293032588010.5435/JAAOS-D-17-00636
- 38Parisien RL, Ehlers C, Cusano A et al. The statistical fragility of platelet-rich plasma in rotator cuff surgery: a systematic review and meta-analysis. Am J Sports Med (2021) 49(12): 3437–34423364688410.1177/0363546521989976
- 39Parisien RL, Danford NC, Jarin IJ et al. The fragility of statistical findings in achilles tendon injury research: a systematic review. J Am Acad Orthop Surg Glob Res Rev (2021) 5(9): e21.00018344919768415978
- 40Parisien RL, Constant M, Saltzman BM, et al (2021) The fragility of statistical significance in cartilage restoration of the knee: a systematic review of randomized controlled trials. Cartilage 13(1_suppl): 147S–155S.
- 41Parisien RL, Trofa DP, O'Connor M et al. The fragility of significance in the hip arthroscopy literature: a systematic review. JBJS Open Access (2021) 6(4): e21.0003534703967854217310.2106/JBJS.OA.21.00035
- 42Parisien RL, Trofa DP, Cronin PK et al. Comparative studies in the shoulder literature lack statistical robustness: a fragility analysis. Arthrosc Sports Med Rehabil (2021) 3(6): e1899–e190434977646868924510.1016/j.asmr.2021.08.017
- 43Pascoal E, Liu M, Lin L et al. The fragility of statistically significant results in gynaecologic surgery: a systematic review. J Obstet Gynaecol Can (2022) 44(5): 508–5143495441110.1016/j.jogc.2021.11.016
- 44Rickard M, Lorenzo AJ, Hannick JH et al. Over-reliance on p values in urology: fragility of findings in the hydronephrosis literature calls for systematic reporting of robustness indicators. Urology (2019) 133: 204–2103137429010.1016/j.urology.2019.03.045
- 45Robinson T, Al-Shahwani N, Easterbrook B et al. The fragility of statistically significant findings from randomized controlled trials in pediatric appendicitis: a systematic review. J Pediatr Surg (2020) 55(5): 800–8043214597610.1016/j.jpedsurg.2020.01.030
- 46Robinson NB, Fremes S, Hameed I et al. Characteristics of Randomized clinical trials in surgery from 2008 to 2020: a systematic review. JAMA Netw Open (2021) 4(634190996824631310.1001/jamanetworkopen.2021.14494
- 47Ruzbarsky JJ, Khormaee S, Rauck RC et al. Fragility of randomized clinical trials of treatment of clavicular fractures. J Shoulder Elbow Surg (2019) 28(3): 415–4223077182610.1016/j.jse.2018.11.039
- 48Ruzbarsky JJ, Rauck RC, Manzi J et al. The fragility of findings of randomized controlled trials in shoulder and elbow surgery. J Shoulder Elbow Surg (2019) 28(12): 2409–24173142022710.1016/j.jse.2019.04.051
- 49Ruzbarsky JJ, Khormaee S, Daluiski A The fragility index in hand surgery randomized controlled trials. J Hand Surg Am (2019) 44(8): 698.e1–698.e73042019710.1016/j.jhsa.2018.10.005
- 50Schröder A, Muensterer OJ, Oetzmann VC Meta-analyses in paediatric surgery are often fragile: implications and consequences. Pediatr Surg Int (2021) 37(3): 363–36733454848790005410.1007/s00383-020-04827-5
- 51Shen C, Shamsudeen I, Farrokhyar F et al. Fragility of results in ophthalmology randomized controlled trials: a systematic review. Ophthalmology (2018) 125(5): 642–6482924174410.1016/j.ophtha.2017.11.015
- 52Skinner M, Tritz D, Farahani C et al. The fragility of statistically significant results in otolaryngology randomized trials. Am J Otolaryngol (2019) 40(1): 61–663047212410.1016/j.amjoto.2018.10.011
- 53Svantesson E, Hamrin SE, Danielsson A et al. Strength in numbers? The fragility index of studies from the Scandinavian knee ligament registries. Knee Surg Sports Traumatol Arthrosc (2020) 28(2): 339–3523119024510.1007/s00167-019-05551-x
- 54Tignanelli CJ, Napolitano LM The fragility index in randomized clinical trials as a means of optimizing patient Care. JAMA Surg (2019) 154(1): 74–793042225610.1001/jamasurg.2018.4318
- 55Shochet LR, Kerr PG, Polkinghorne KR The fragility of significant results underscores the need of larger randomized controlled trials in nephrology. Kidney Int (2017) 92(6): 1469–14752875455110.1016/j.kint.2017.05.011
- 56Docherty KF, Campbell RT, Jhund PS et al. How robust are clinical trials in heart failure. Eur Heart J (2017) 38(5): 338–34527742808
- 57Chase KB, Matt VB Unbreakable? An analysis of the fragility of randomized trials that support diabetes treatment guidelines. Diabetes Res Clin Pract (2017) 134: 91–10510.1016/j.diabres.2017.10.007
- 58Adie S, Harris IA, Naylor JM et al. The quality of surgical versus non-surgical randomized controlled trials. Contemp Clin Trials Commun (2017) 5: 63–662974062110.1016/j.conctc.2016.12.001
- 59Maggard MA, O'Connell JB, Liu JH et al. Sample size calculations in surgery: are they done correctly?. Surgery (2003) 134(2): 275–2791294732910.1067/msy.2003.235
- 60Ioannidis JPA The proposal to lower p value thresholds to 005. JAMA (2018) 319(14): 1429–14302956613310.1001/jama.2018.1536
- 61Lydersen S, Pradhan V, Senchaudhuri P et al. Choice of test for association in small sample unordered r x c tables. Stat Med (2007) 26(23): 4328–43431731122010.1002/sim.2839
- 62Yusuf S, Peto R, Lewis J et al. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis (1985) 27(5): 335–371285811410.1016/S0033-0620(85)80003-7
- 63Bomze D, Asher N, Hasan AO et al. Survival-inferred fragility index of phase 3 clinical trials evaluating immune checkpoint inhibitors. JAMA Netw Open (2020) 3(1033095247758493010.1001/jamanetworkopen.2020.17675
- 64 ESCP EAGLE Safe Anastomosis Collaborative. (2021) ESCP Safe Anastomosis ProGramme in CoLorectal SurgEry (EAGLE): study protocol for an international cluster randomised trial of a quality improvement intervention to reduce anastomotic leak following right colectomy. Colorectal Dis 23(10): 2761–2771. doi:https://doi.org/10.1111/codi.15806
- 65Andrade C The use and limitations of the fragility index in the interpretation of clinical trial findings. J Clin Psychiatr (2020) 81(2): 20f1333410.4088/JCP.20f13334
