1 INTRODUCTION

Drug development programs typically include exploratory (Phase II) and confirmatory (Phase III) randomized controlled trials (RCTs) to assess the efficacy, safety and appropriate dosages of an experimental (new) treatment. For regular “large disease” drug development programs decisions to conduct a Phase III trial are based on positive Phase II trials. If these trials are only retrospectively evaluated in combination, that is, during the drug marketing authorization request, the ad hoc synthesis may induce a form of decision-induced bias (the succeeding trials are only conducted when the first trials were positive). Such a bias is not an issue if the early and late Phase trials are prospectively considered in the design phase (eg, a seamless approach).

However, it is not uncommon that in rare diseases, no more than two independent RCTs are conducted and available, one exploratory and one confirmatory.¹ Phase II primary endpoints are typically biomarkers or surrogate outcomes.² Phase III primary clinical outcomes are likely established endpoints and they may either require (1) larger sample sizes, (2) more costly collection, (3) to be observed after a considerable time, or (4) be more variable outcomes than early-phase outcomes, therefore, even if N = N₂ + N₃ number of patients participate in both trials, only N₃ patients will be available to provide responses for the primary clinical outcome of interest. Biomarkers (early-phase) and secondary clinical outcomes are often observed earlier and, therefore, easily included in both trials and, hence, available for all N patients. After both trials have been conducted, inference on the treatment efficacy is typically performed by evaluating the late-phase outcome responses of N₃ patients. In a rare disease setting, N₃ may not be large enough to solidly confirm treatment efficacy. In assessing the totality of evidence, the positive results from the Phase II trial could retrospectively be seen as supportive, even if the two clinical trials were designed/conducted independently, as typically the early-phase outcome would be assumed to be associated with the late phase primary clinical outcome. Throughout the article the terms “retrospective (ly)” denote the retrospective combination of the available Phase II and Phase III trial after both trials are completed and their final results are available.

For example, Galafold (migalastat) acquired marketing authorization as an orphan drug for the treatment of Fabry disease in 2016 within Europe. Fabry disease is a rare, progressive disorder with an estimated prevalence of 1:117 000 to 1:40 000.³ The condition affects major organs and may result in life-threatening events. Until then, standard treatment for Fabry disease consisted of Enzyme Replacement Therapy.³ Two main studies were submitted during the marketing authorization of migalastat; one randomized, placebo-controlled (AT1001-011, migalastat vs Placebo) superiority study and one active comparison randomized trial (AT1001-012, migalastat vs Enzyme Replacement Therapy), with a noninferiority design.

In trial 011 patients switched to migalastat 6 months postrandomization, while in trial 012 primary follow-up was considerably longer, with switching taking place 18 months postrandomization. In the first trial, the change in average globotriaosylceramide (GL-3) inclusions from baseline to 6 months was the primary outcome which produced nonconclusive evidence. The second trial utilized the annualized change in glomerular filtration rate (eGFR) at month 18 as primary clinical outcome (Table 1). Both GL-3 and the annualized change in eGFR at month 6 were collected in both trials (011 and 012). No strong correlation has been established in the literature between the GL-3 outcome and the change in glomerular filtration rate (eGFR).⁴

In study 011 after 6 months of treatment with migalastat 150 mg, eGFR values increased, whereas in the placebo treated group eGFR values declined.³ This outcome among other secondary results led to the conduct of study 012. In trial 011, all patients treatment switched to migalastat at 6 months, an action that restricts the observation of a treatment effect on the primary late-phase outcome. Given the limited available data, evidence from both trials were retrospectively (ad hoc) assessed for the final approval decision.

Analysis methods that use the relation between early and late-phase outcomes may be applied to retrospectively, but formally, synthesize the evidence on treatment efficacy across the two trials. Engel and Walstra⁵ formulated a double-regression (DR) approach, which can aid in more precise treatment effect estimation, by accounting for unobserved late-phase outcome responses via observed early-phase outcome responses. Their method utilizes the correlation to ultimately inform the mean and variance estimates of the treatment effect on the late-phase outcomes. For large samples their method has the potential to increase precision. However, for small sample sizes this is not necessarily true.⁶ Previously, in RCTs the DR approaches have been suggested mainly to inform treatment selection during interim analysis in seamless Phase II/III designs.^7-9 Double-regression methods can be even generally applied wherever there is possibility to include early outcome information in decision making during the course of a trial.¹⁰

A Bayesian double-regression (BDR) analogue can be readily constructed¹¹ which maintains similar limitations to the frequentist alternative but could flexibly model the two Phase III outcomes' data. Such a model can include historical trial data (ie, Phase II early-phase outcome data or external information on the early and late-phase outcome correlation) as a elicited prior distributions.¹² Furthermore, this Bayesian model accounts for the uncertainty around each parameter during the borrowing of information.

In this article, we investigate how to model and estimate the efficacy of a new treatment on the late-phase clinical outcome, using data on early-phase outcomes from both trials. Most literature on double-regression focuses on design aspects such as interim analysis or seamless design of phase II/III trials, though, in the present article we propose methods that would be applied retrospectively (ad hoc) only after the Phase III trial. We propose and investigate methods that either account or do not account for the potential decision-induced bias when combining retrospectively the Phase II and Phase III trials. We investigate the two proposed models, the bias corrected DR approach and the flexible Bayesian approach regarding their performance to estimate the treatment effect on the late-phase outcome. We focus on two related key problems: (1) the magnitude of the type 1 error inflation when retrospectively combining data from Phase II and III and (2) how to estimate the treatment effect on the late-phase outcome, using results from both studies and we assess this estimate in terms of bias and variance.

The article is organized as follows. First, we describe a bivariate linear model, we introduce its conditional form and we formalize the (often visual) retrospective pooling by utilizing DR with nonavailable Phase II late-phase outcome data, then briefly discuss specific model variations, for example, the single-regression (SR) approach. We introduce the problem of decision-induced bias moving from Phase II to Phase III based on the Phase II early-phase outcome in Section 3 and then provide an approximate analytical solution. In Section 4, we propose and formulate a Bayesian two-step solution to the estimation problem, a model that down-weights the impact of the biomarker data via a historical power prior. This prior dynamically accounts for the bias in estimating the same treatment effect across the two trials, by accounting for additional between-trial differences (variability) around the biomarker outcome effect. The article ends with a discussion and steps for further research.

2 MODELS FOR THE JOINT PHASE II AND III DATA

Consider a Phase II trial of total sample size N₂ and a Phase III trial of total sample size N₃. For both trials it is assumed that a number of patients (N_k = n_ck + n_ek,  n_k = N_k/2,  k = 2, 3) are randomized to the control and experimental treatment. Let us denote Y_ik the late-phase treatment response for patient i in trial k and X_ik the early-phase treatment response for patient i in trial k, k = 2, 3,  i = 1, 2, … N_k.

2.2 Double regression to estimate the effect of primary late-phase outcome

At the end of both trials early-phase outcome data X for N = N₂ + N₃ patients and late-phase outcome data Y₃ for only N₃ patients are observed. As Y₂ is not observed, Y = Y₃ and X = (X₂, X₃) now denote the observed late-phase and early-phase outcome data which correspond to patients of Phase II and Phase III trials. Y corresponds to the outcome of interest related to which estimation and hypothesis testing will be performed in N₃ patients. The DR utilizes the relation between early-phase and late-phase outcomes and allows estimation of the main parameter of interest, the treatment effect on the late-phase clinical outcome, b_y (Figure 1).

Based on the DR method, parameters a_x, b_x, and ${\sigma }_x^2$ are estimated via the regression of X|t on N patients, as ${â}_x,{\widehat{b}}_x$, $s_x^2$ and parameters $a_0,b_0,\gamma$, and ${\sigma }_0^2$ are estimated via the regression of Y₃|X₃, t on N₃ patients, as ${â}_0,{\widehat{b}}_0,\widehat{\gamma }$, $s_0^2$, while $s_y^2=s_0^2+{\widehat{\gamma }}^2{s}_x^2$, ${â}_y={â}_0+\widehat{\gamma }{â}_x$, $\widehat{\rho }=\widehat{\gamma }{s}_x/s_y$.^{5, 8} The primary effect of interest b_y is then estimated as:

$$\begin{array}{ll}{\widehat{b}}_y={\widehat{b}}_o+\widehat{\gamma }{\widehat{b}}_x.& \end{array}$$ (eq1)

The variance of ${\widehat{b}}_y$ is shown in Reference 5 to be equal to

$$\mathrm{var}({\widehat{b}}_0)+{\widehat{\gamma }}^2\mathrm{var}({\widehat{b}}_x)+{\widehat{b}}_x\mathrm{var}(\widehat{\gamma })+2{\widehat{b}}_x\text{cov}({\widehat{b}}_0,\widehat{\gamma })$$

estimates of the above can be obtained by using the individual estimates acquired from the regression analyses (m2). Under model (m2), hypothesis testing is performed as H₀ : b_y ≤ 0 vs H₁ : b_y > 0 via $z_{1-{\alpha }_3}>{\widehat{b}}_y/\sqrt{\widehat{\mathrm{var}}(b_y)}$, where $z_{1-{\alpha }_3}$ is the $(1-{\alpha }_3)th$ standard normal quantile and ${\alpha }_3$ is the alpha level of the late-phase primary outcome of phase III trial. A direct Bayesian analogue to the conditional model (m2) has been discussed elsewhere.¹¹ Under diffuse “non-informative” priors, this Bayesian model has been shown to produce comparable posterior means for all parameters to the estimates produced by model (m2).

3 TYPE 1 ERROR INFLATION AND BIAS DUE TO SELECTION BASED ON EARLY-PHASE OUTCOME RESULTS

As we observe in (eq3), the inflation in MSE depends on (i) the decision threshold to initiate the Phase III trial through $\lambda$ parameter, (ii) the Phase II early-phase outcome mean (${\mu }_{x2}$) and variance $(2{\sigma }_{x2}^2/n_2$), (iii) the number of patients in the Phase II trial (n₂) and (iv) and the magnitude of the correlation ($\rho$). An increase in ${\sigma }_{x2}$ results in an increase of MSE, while as n₂ decreases, the MSE increases as well. A similar behavior is observed in terms of Type I error (Figure 2). More specifically, Type I error rates increase considerably with higher $\rho$, while the power curves, in general, increase with more patients being allocated to the Phase III trial (n₃) (Figure 2).

The above expressions hold when treatment arms within studies are equal. Nonetheless, similar analytical expressions for unequal within study allocation ratios, can be acquired by appropriately changing the variances of ${\widehat{b}}_{x3},{\widehat{b}}_{x2}$ in Appendix B.1 based on the treatment arms sample sizes. For example, if the allocation ratio between arms in the Phase II trial equals to 1:2, then the Phase II early-phase endpoint variance increases, $9{\sigma }_x^2/2N_2$ and the introduced bias could be reduced by half.

4 BIAS REDUCTION BY ACCOUNTING FOR BETWEEN-TRIAL EARLY-PHASE OUTCOME VARIABILITY

All models above, including the bias corrected model, assume that the true overall treatment effect remains common between trials, no between-study variability on the early and late-phase outcomes exist and therefore, all N observations are derived from the same population. Phase II vs Phase III trials typically do not have similar protocols, as the Phase II trials are usually more restrictive in patient inclusions, therefore, exploring between-study variability becomes relevant.

The decision-induced bias discussed in Section 3, would materialize as difference in treatment effects between the two available trials as well. Therefore, accounting for between-study variability may act as a less rough approach to minimize this decision-induced bias. A proper estimation of the between-trial early-phase outcome variance is not feasible with just two available studies,^14-17 therefore, in this article we choose not estimate but only account for this variance to aid towards the reduction of the bias.

To achieve this, we utilize a mechanism based on power priors to account for the between-study differences within a Bayesian framework.¹⁸ By estimating a power parameter $\widehat{\eta }$ that represents conflict between the early-phase outcome data of the two available trials, model (m3) can be further extended to account for the early-phase outcome effect excess between-trial variability, along with any other biases.^18-20

5 SIMULATION STUDY

The main four approaches discussed are summarized in Table 2. The corrected double-regression approach as shown in Section 3 can be considered a rough (approximate) approach to minimize the decision-induced bias. The Bayesian flexible double-regression approach minimizes this bias by accounting for between-trial differences without ad hoc corrections. Their relative performance in the analysis of the Phase III late-phase outcome data, also in comparison to the two more trivial approaches (single and double-regression) is the main focus of the simulation study.

For illustrative purposes, we assume that the two available Phase II and Phase III trials had a similar control treatment, therefore, the Phase III trial would have been designed as a placebo-controlled trial. In this section, we assume that the decision to conduct the Phase III trial was taken on the basis of available evidence in the first Phase II trial on a single early-phase outcome. At the end of the Phase II trial, individual data of N patients are available on the early-phase and data of N₃ are available on the late-phase outcomes. The simulation study results were derived from a bivariate normal model simulation strategy as described in Appendix C.

The SR, DR, DRC methods ignore any between-study variability and therefore assume a different underlying data generating model in comparison to the Bayesian flexible double-regression (BFDR) approach. Even though, they are not directly comparable (Table 2), we empirically compared the four aforementioned statistical methods by generating at least 10 000 simulated combinations of the two available trials data. To do so, we simulated scenarios of the final trial analysis on the late-phase primary endpoint assuming a variety of combinations between the early-phase (b_x) and late-phase (b_y) outcome treatment effects. The latter were varied as (Scenario I) b_y = b_x = 0, (Scenario II) b_y = b_x = 0.6, (Scenario III) b_x2, b_y2 = 0,  b_x3, b_y3 = 0.2, and (Scenario IV) b_y = 0.6, b_x = 0, we assumed that $\rho =0.9$, ${\alpha }_2\in \{0.05,0.1,0.2\}$ the alpha level of the early-phase primary outcome of Phase II trial, while all within-study variances were set equal to 1. In the simulation setup we introduce a simulative parameter that place additional between-trial variance on the early-phase (${\tau }_x$) and late-phase (${\tau }_y$) outcomes (see Appendix C for details). Specific alternative versions of scenarios I and II were produced by varying $\rho$ and ${\tau }_y,{\tau }_x$.

The first (I) scenario describes variations of the strict null (${\tau }_y={\tau }_x=0$) and null hypothesis with additional between-trial variance (${\tau }_y={\tau }_x=0.3$), while the second (II) scenario describes a common alternative hypothesis on both outcomes and trials. Scenario III can occur when heterogeneous populations are selected for the Phase II and Phase III trial, while the fourth (IV) scenario describes a situation where the late-phase outcome true effect exists but the early-phase outcome equals to 0. All remaining settings (ie. number of trials (k), total sample sizes N, sample size ratio between trials N₂ : N₃, within-study allocation ratios n_ck : n_ek) were reflective of a typical rare disease setting and based on the Galafold example (Table 1). All simulations were performed via R²³ and JAGS.¹³

6 DISCUSSION

In a drug development procedure, it is not uncommon that positive Phase II results on early-phase (biomarker) outcomes are not predictive of a Phase III success on late-phase clinical outcomes. If Phase II and Phase III results are then assessed (perhaps informally) jointly to support efficacy, this retrospective (ad hoc)) assessment may be subject to decision-induced bias and may increase uncertainty of the true primary late-phase treatment effect. Such an informal combination of results may increase to a great extent (more than three times) the Type I error rate of null hypothesis, rendering the retrospectively combined late-phase true treatment effect misleading. Especially in rare diseases, where the validation of early-phase surrogate endpoints can become problematic, due to the small and often heterogeneous populations, the small sample sizes and the insufficient number of available trials, only late-phase hard endpoints are usually appropriate to prove treatment efficacy.

In this article, in addition to identifying and investigating the above issue, we explored methods that can be utilized in order for early and late Phase trial data to be combined retrospectively (ie, right before drug marketing authorization request), while accounting for the underlying decision-induced bias. The flexible BDR includes the borrowing of historical information, while this model downgrades the historical prior upon early-phase outcome data conflict. The DRC method approximately corrects the biased late-phase mean effect and variance estimate.

In most scenarios, the DRC method better controls the Type I error and bias than the DR and BFDR methods. This is not observed in scenario Ia, where the BFDR controls better the Type I error than the DRC. This possibly happens because the BFDR approach completely downgrades the impact of Phase II trial when its early-phase treatment effect is different than the Phase III trial early-phase treatment effect. Therefore, on average the Bayesian approach becomes less prone to false-positive results based on possible very positive Phase II early-phase outcome trial effects when ${\tau }_x$ is low and/or $\rho$ is high (see, black dots of inner right panel of Figure 2). On the contrary, the DRC corrects the Phase II effect and then utilizes both Phase II and Phase III effects without heavily downgrading the Phase II results data upon data conflict. The DRC requires a known ${\alpha }_2$ but despite being approximate, it applies a more direct (decision-based) penalty to the Phase II effect than the Bayesian approach; which could explain its overall better performance in the simulation.

Both the BFDR and the DRC methods would be an attractive solution to the increased Type I error of the informal retrospective combination of two small available trials. The consideration of these methods was shown to be rather important when, (i) the preceding Phase II trial conservatively (ie, alpha level was small) resulted to the Phase III trial and/or (ii) the association of utilized early and late-phase outcomes is high. An informal combination of results across Phases often happens when both of the above hold, though, when neither holds then the complexity of suggested methods may outweigh the gains of their application.

Alternative versions of the BFDR model could be developed and they may perform more optimally in comparison to the current (ie, in terms of controlling the overall type I error) when applied on the flexible BDR via the use of an alternative guided value.^18-20 The power parameter is imposed on the early-phase endpoint and only indirectly affects the primary late-phase endpoint, therefore, inference on the late-phase endpoint via alternative guided values on the early-phase endpoints could be expected to be more comparable to some extent.

An alternative approach that controls type I error on the late-phase outcome, while borrowing historical information, may also provide a more formal solution.¹⁹ Future research could compare these alternatives vis-á-vis each other or with other methods. More covariates could be included, and then their performance could be tested with ease as all presented models are readily generalizable to full regressions. In this article, we set independent informative priors on the model parameters, however, accounting for the correlation between these parameters could also be considered through a well-defined informative Wishart prior on the whole covariance matrix. Finally, in this work, we accounted for but did not estimate between-study variance. Due to the only two available studies, a proper estimation of the between-study outcome variability is currently known to be almost nonfeasible.^14-17

In the motivating example we assumed that both trials were superiority trials, while if we had kept the initial designs, different strategies may have been more appropriate. Nonetheless, examples of two superiority trials, one Phase II and one Phase III, exist in the literature. For example, the drug development program of thalidomide for the treatment of multiple myeloma contained two randomized superiority clinical studies of similar design, a supportive (GISMM2001) and a main study (IFM 99-06), that compared melphalan-prednisone (control treatment) to thalidomide (experimental treatment).² The supportive study was shorter and it reported clinical response rates and event free survival as primary endpoints. The main study was longer in duration and it reported overall survival, as main endpoint and clinical response rates and event free survival, as secondary endpoints. The suggested methodology could be tailored to account for the possibility of decision-induced bias under survival and other types of outcomes and even to combine different study designs.

Throughout the article normality was assumed, an assumption that could be challenged with rare diseases sample sizes.^1-3 We approximated a truncated normal with a normal distribution with mean and variance equal to that of the former. This decision was made to aid calculations on the distribution mixture (Appendix B). Better approximations for the truncated normal distribution may exist, such as the chi-square distribution and their performance could be explored as well.²⁴ We should note that for moderately sized N₂ in comparison to N and small correlation between the two outcomes, a SR might be more efficient than a DR, due to the noise introduced by the early-phase outcome.⁵ In the simulation study we assumed that the Phase II trial had equal allocation between trial arms, while the Phase III trial had allocation equal to 1:2 between the control vs treatment arm. We expect that our findings would be comparable under different allocations between arm sample sizes, though further investigation could provide more insights between the relative performance of BDR and DRC methods.

In this article, we performed a post hoc (retrospective) combination of available information after the conduct of the Phase II and Phase III trial. However, it may be very relevant to (prospectively) plan to pool the data from both studies and to use the early-phase outcomes of the Phase II study to increase the precision, with which the efficacy on late-phase outcome is estimated overall.^7-9 An alternative strategy could be to conduct one single trial with interim analysis, then, based on the observed treatment effects on the early-phase endpoints decide whether to follow-up the patients.⁸

To conclude, especially in a small population context, the often informal retrospective pooling of a single Phase II early-phase outcome data to support the true late-phase outcome data inference at the end of a single confirmatory Phase III trials could induce bias and it should be performed via formal numerical approaches. Such approaches should control this decision-induced bias, in order to avoid inflating the Type I error under the null hypothesis and prevent overestimating our beliefs on the primary treatment effect. We hope that this article, except for introducing possible solutions, raises awareness of potential mishaps with post hoc combinations of trial outcome results.