Synthetic control arms in studies of multiple myeloma and diffuse large B-cell lymphoma

Within the past year, several publications have compared the outcomes of clinical trials in relapsed/refractory (R/R) diffuse large B-cell lymphoma (DLBCL) or R/R multiple myeloma (MM) against retrospective real-world datasets.^1-4 Two such datasets include the SCHOLAR-1 cohort in DLBCL (comprising 636 patients analysed between 2001 and 2014) and the MAMMOTH cohort in MM (comprising 275 patients analysed between 2017 and 2018).^{5, 6} When these observational datasets are compared against single-arm trials in such a manner, their use has been described as constituting a ‘synthetic control’ arm.^{7, 8}

However, synthetic control arms may be inadequate substitutes for true control arms generated via randomised controlled trials (RCTs). Specific critiques of SCHOLAR-1 and MAMMOTH as synthetic controls involve differences in patient eligibility due to (i) slight but significant variations in definitions of R/R status and (ii) general exclusion of unfit patients from trials versus their general inclusion in observational datasets.^{9, 10} Furthermore, the generalisability of historical observational results as controls for more recent trials is uncertain. Lastly, whether potential conflict of interests exist in these analyses has not been described.

We thus characterised all studies using SCHOLAR-1 or MAMMOTH as synthetic controls by comprehensively reviewing relevant publications or abstracts available on PubMed or Google Scholar before the data cut-off of 30 September 2021. As shown in Fig 1, a total of four studies in DLBCL^{3, 4, 11, 12} and five studies in MM^{1, 2, 13-15} were included in our present analysis. These included four peer-reviewed publications and five conference abstracts (Table I).^1-4

In DLBCL, the novel-agent arm in all four studies enrolled patients during more recent intervals than the SCHOLAR-1 dataset.^{3, 4, 11, 12} All four studies adjusted for key potential confounders using individual patient data. However, one study did not adjust for differences in the specific definitions of R/R status in SCHOLAR-1 versus the JULIET trial of tisagenlecleucel.^{5, 12, 16} Similarly, one study did not adjust for slight differences in clinical response definitions between the TRANSCEND trial of lisocabtagene maraleucel (Lugano criteria) versus SCHOLAR-1 (original International Working Group 2007 criteria).^{4, 17, 18}

Three of four DLBCL studies reported improved overall survival (OS) with the novel agent.^{3, 4, 11} In the fourth study, the authors acknowledged a statistical difference in OS but did not report further details because they felt that this incremental improvement was clinically insignificant.¹² Of note, this negative study did not report any potential conflicts of interest. Among the three studies that showed a benefit to the novel agent, all three reported potential conflicts of interest such as funding or medical-writing assistance by the manufacturer of the novel agent.

In MM, the novel agent arms in all five studies were contemporaneous with the MAMMOTH dataset and analyzed similar patient cohorts.^{1, 2, 13-15} However, unlike the MAMMOTH dataset, the DREAMM-2 trial of belantamab mafodotin included treatment intolerance as a determinant of R/R status.^{6, 19} This variation in the definition of R/R status was not commented upon in the corresponding synthetic control analysis.¹³ Finally, while two studies used regression modeling to compare arms, they did not report any statistical adjustments based on individual patient data.

All five studies in MM reported significantly improved OS with the novel agents. Two studies noted a comparative benefit in OS but no corresponding benefit in progression-free survival.^{13, 15} Similarly, three studies did not find any significant improvement in overall response rates with the novel agent.^{1, 13, 15} All five studies reported potential conflicts of interest, most typically in the form of authorship by employees of the manufacturer of the novel agent.

In our present review of studies using the SCHOLAR-1 and MAMMOTH datasets as synthetic control arms, we thus noted several potential items of concern. Firstly, several studies (primarily in DLBCL) compared more recent trials against older control arms. Secondly, statistical adjustments using individual patient data were employed in most but not all studies. Thirdly, most studies reported potential conflicts of interest with the pharmaceutical manufacturers of the novel agent being analysed. In contrast, the only study without any conflicts of interest was also the only study to determine that the novel agent in question did not meaningfully improve OS.¹²

In addition to these specific issues raised by our present study, broader issues with synthetic control arms include the presence of unmeasured confounders. For example, patients with significant comorbidities are generally included in registries but excluded from trials.⁹ Similarly, patients in registries only need to be alive at the time of eligibility; in contrast, clinical trials only characterise patients who survived until treatment initiation. This, in turn, may select for more indolent disease among clinical trial participants versus synthetic controls.²⁰ Landmark analyses can potentially correct for these differences in ‘time zero’ but were not performed in any of the studies we identified. Lastly, performing multiple post hoc comparisons against the same static dataset can raise the risk of generating false-positive signals.^{21, 22}

Limitations of our present study include our specific emphases on SCHOLAR-1 and MAMMOTH and not any other observational datasets. We were not able to determine how much study results were meaningfully influenced (if at all) by differences in years of enrolment, R/R status, or the presence of potential conflicts of interest. Lastly, regarding the ZUMA-1 trial of axicabtagene ciloleucel, the use of SCHOLAR-1 as a synthetic control may have been pre-planned in ways that we are unable to fully characterise.²³

Nevertheless, when true RCTs are not feasible, we suggest that appropriate steps be taken to ensure the validity of synthetic control arms. For example, as is being done with the observational KarMMa-RW dataset versus the KarMMa trial of idecabtagene vicleucel,²⁴ a contemporaneous study-specific control arm can be compiled prospectively (even if not randomised) to reflect the changing status quo more accurately for standard-of-care therapies. Additionally, observational datasets should be governed by neutral co-operative groups and specialty societies to encourage pre-planned comparisons and to minimise conflicts of interest.

In summary, our present findings highlight several problems with the use of observational datasets as synthetic control arms in malignant haematology. While these analyses may provide additional context regarding novel therapeutic agents, they cannot serve as replacements for RCTs.

Author contributions

Aaron Goodman, Vinay Prasad, and Ghulam Rehman Mohyuddin conceptualised the study. Rahul Banerjee, Shonali Midha, and Amar H. Kelkar performed the research. All authors analysed the data and contributed substantially to writing the manuscript.

Conflict of interest

Rahul Banerjee reports the following disclosures: Guidepoint Global (consulting), Pack Health (research), Sanofi (consulting), SparkCures (consulting). Aaron Goodman reports the following disclosures: EUSA Pharma (consulting), Seattle Genetics (consulting). Vinay Prasad reports the following disclosures: Arnold Ventures (research), Johns Hopkins Press (royalties), United Healthcare (consulting), Evicore (honoraria). The remaining authors do not have any conflicts of interest to report.