Critical appraisal tools and rater training in systematic reviews and meta-analyses

EDITOR—We appreciate the opportunity to respond to the letter by Rehm and Thahir¹ on our systematic review and meta-analysis.² First, the search strategy, developed by the researchers and supported by an information specialist, is in accordance with systematic review methodology and was transparently reported. The team created extensive and multiple terms – found via different approaches – to reduce bias, and truncation operators were used to increase sensitivity.

Presently, there is no criterion standard tool for assessing the methodological quality in non-randomized studies for systematic reviews. Although the Newcastle–Ottawa Scale (NOS) is widely used and recommended by the Cochrane Collaboration, its lower interrater reliability has been recognized.^{3, 4} Nevertheless, Oremus et al.⁴ concluded that standardized rater training before assessment may improve agreement. In our study, three trained independent reviewers assessed the quality after piloting the scale.

We fully agree that including randomized controlled trials (RCTs) would have lowered meta-analysis’ bias. In cases in which RCTs are not ethical nor possible, as with the topic of our research, case–control studies may be the best alternative. We agree that the selected controls in the case–control studies should be representative for their population. Therefore, we assigned a low NOS score for selection and definition of controls, see Table S5 in Van der Looven et al.²

Furthermore, as mentioned in detail in our limitation section,² we acknowledge the presence of source bias of the included studies. Gutman et al. suggest that a time-dependent bias and selection bias, among other factors, could be responsible for the differences in diagnoses between different databases.⁵ However, given that the diagnosis of neonatal brachial plexus palsy (NBPP) is clinically obvious directly after birth, our meta-analysis is not likely to have been affected by this time-dependent factor. The Kids Inpatient Database is not subject to selection bias by a clustering effect. Moreover, indexed with the International Classification of Disease codes, it enables database comparison. We agree that the location and time distribution of the studies in our review is quite wide. However, the intent of our systematic review was to combine all available worldwide data from a predefined timespan to clarify our research question.

Additionally, we feel the critical appraisal by Rehm and Thahir¹ of our sensitivity analyses is quite flawed. The analyses were performed post hoc to clarify, not to decrease, the heterogeneity. The authors did not remove studies from the meta-analysis nor manipulate eligibility criteria.

Rehm and Thahir¹ also focus strongly on their doubts about the validity of the NBPP incidence decrease. Again, in our limitation section we clearly state that the additional incidence assessment is prone to selection and source bias as it was not the primary aim of the meta-analysis.² Weighing of the different study data, including the Swedish registry data, was correctly applied to minimize bias for the incidence calculation. Overlapping study populations were systematically identified and only included once, as appropriate.

We agree critical appraisal is an essential element of improving the reporting of data, therefore we appreciate the concerns of Rehm and Thahir.¹ However, meta-analyses are complex studies that include a series of decisions to be made which can create room for subjectivity and, with that, critique. Therefore, a well-documented reporting practice, as recommended by the PRISMA statement, was transparently presented.