Use of Stabilized Inverse Propensity Scores as Weights to Directly Estimate Relative Risk and Its Confidence Intervals

Introduction

Observational studies have been used by medical researchers seeking to make inference on the effect of treatments on outcomes. Compared with those in randomized clinical trials, participants' characteristics in an observational study may not be balanced between treated and untreated groups. Consequently, the estimate of a treatment effect may be biased without appropriate adjustment when receipt of treatment is dependent on patients' characteristics (confounders) that also are associated with outcomes. Propensity scores were introduced by Rosenbaum and Rubin [1,2] and have been used by many researchers to obtain the treatments effects in observational studies [3–8]. A propensity score is the probability of receiving treatment given a set of known covariates and can be used to balance covariates between treated and untreated to obtain an unbiased estimate of treatment effects. Typically, propensity scores in an observational study can be obtained from ordinary logistic regressions if the treatment is binary.

The simplest use of propensity scores is that they can be included as covariates in outcome modeling. One can first fit a propensity score model that includes many potential covariates, and then the outcome model only has to include the propensity score and a few covariates that have no association with treatment [3,9]. But this approach can perform poorly if the sample linear discriminant based on covariates is not a monotone function of propensity score [1]. There are three additional strategies that use propensity scores to reduce selection bias: matching, stratification, and inverse probability of treatment weighting (IPTW). Matching subjects in treated groups with those in untreated groups with similar propensity scores can balance the known covariates and reduce selection bias. But it can also result in significant loss of observations of treated subjects, particularly if the untreated pool is small. Stratification places subjects into several mutually exclusive groups or strata. Based on their propensity scores, treatment effects are estimated from each stratum and averaged across strata to estimate the overall treatment effect [3,10]. The limitation of stratification is that one overall treatment effect may not be interpretable when the treatment effects of strata are very different in scale especially in direction. In addition, subjects in different strata may not separate into distinguishable groups that are meaningful to clinicians. The third propensity score approach is to use IPTW-weighted estimators to obtain treatment effects adjusting for known confounders [6,11,12]. This approach can incorporate time-dependent covariates and deal with censored data and produce one overall estimate of treatment effect.

For continuous outcome variables, there are three unbiased estimators for treatment effects [10,12] based on the IPTW, which have shown consistency but with different variance estimators. Nevertheless, these variance estimators are large-sample based and may produce large variance estimates and decrease efficiency of the estimators [10]. Estimators and variance estimates are less developed for discrete outcome variables. Accurate variance estimation of the treatment effect is critical to testing hypotheses. Underestimation of the variance produces inappropriately narrow confidence intervals and leads to falsely rejecting the null hypothesis. In addition to the large-sample-based variance estimators, others have suggested the use of the bootstrap method to obtain the variance of treatment effects [13,14], which can be used for medium or large samples and for different effect measures, for example, difference for continuous outcomes, incidence rate ratio for count data, and odds ratio for dichotomous outcomes. Nevertheless, the bootstrap method is not suitable for small data sets because there are few values to select from and it involves complex programming [15,16]. A robust variance estimator [13,17,18] has also been used to obtain standard error of the treatment effect. This approach adjusts for the lack of independence in replications of records for a subject in the pseudo data and is available in common statistical software packages such as the SAS PROC GENMOD (SAS Institute, Cary, NC). There are also a variety of weights developed based on sampling designs in survey studies to accurately compute estimates of population statistics and their standard errors from a small sample [19].

The aim of this study was to evaluate the use of stabilized weights (SWs) to obtain directly from conventional regression in observational studies both the treatment effects and their appropriate confidence intervals in the presence of confounders. In addition, we provide some comparisons of type I error rates using SWs to the robust variance estimator.

Statistical Methods

Let z be an indicator of binary treatment with 1 for treated and 0 for untreated, X be a row vector of confounders for the probability of treatment and outcome, π be the propensity score, and y be the outcome variable. Suppose that there are N subjects in a data set, with n₁ subjects who received the treatment and n₀ subjects who did not, N = n₀ + n₁. The probability of treatment without considering covariates is p = n₁/N, and the probability of no treatment is 1 − p. The propensity score π_i = prob (z = 1|X_i) is the probability of treatment given the observed covariates X_i. The propensity score can be estimated with a logistic regression model , where β is a vector of parameters to be estimated from data. With the covariates X in the propensity score model using IPTWs as weights, if z_i = 1 and if z_i = 0, where W_i denote the IPTW for subject i.

In the pseudo data using IPTWs, the number of observations is the sum of weights

N_w is always greater than N, the sample size of the original data. To examine this further, assume that there is only one covariate, x₁, which is dichotomous and associated with the probability of being treated with a coefficient β_x1z. For subjects with x₁ = 0, let m₁ be the number of treated subjects and m₀ be the number of untreated subjects, M = m₁ + m₀, and e₀ is the probability of being treated when x₁ = 0. For subjects with x₁ = 1, let l₁ be the number of treated subjects and l₀ be the number of untreated subjects, and L = l₁ + l₀, and e₁ is the probability of being treated when x₁ = 1. The sample size of the pseudo data with IPTWs is

(1)

where e₀ and e₁ are estimated from data, and . Substituting and into Equation (1),

(2)

Thus, the sample size doubles in the pseudo data. This is also true when there are other categorical variables that are associated with the probability of being treated. Consequently, regression estimates with IPTWs tend to reject the null hypothesis too frequently because of inflated sample sizes.

An improvement to the IPTW is the use of SWs. SWs have been proposed in modeling time-varying treatment status in reducing selection bias in observational studies [20,21]. The purpose of using SW in these studies is reducing the weights of either those treated subjects with low propensity scores or those untreated subjects with high propensity scores. For this article, we only considered constant treatment status, if z_i = 1 then , and if z_i = 0, then , where p is the probability of treatment without considering covariates. We will show that the use of SW reduces the type I error by preserving the sample sizes in pseudo data sets. Again, assuming that there is only one dichotomous predictor for the probability of being treated, x₁, p can be estimated from data as . Using the SWs,

(3)

Equation (3) demonstrates that using SWs in observational studies will result in a pseudo data with sample size that is the same as that of the original data. Thus, the variance estimate of treatment effect is appropriate directly from conventional regression with SWs. This is also true when other categorical variables that are associated with the probability of being treated exist. The impact of continuous variables on sample size in the pseudo data cannot be revealed in closed forms and will be evaluated by simulations in the next section.

Simulation Studies and Results

The simulations were designed to evaluate the use of SWs to estimate the effect of treatment and its variance in the presence of confounders and to obtain appropriate confidence intervals using conventional regressions analyzing data from observational studies. Specifically, we examined the sample sizes in the pseudo data sets and type I error rates when confounders in the propensity score and outcome models were dichotomous, categorical, and continuous.

Simulation Algorithm

Probability model for treatment, z. The treatment indicator variable, z, was simulated according to model (4)

(4)

where α is the intercept and is equal to 0.69 and X is a row vector of dichotomous, categorical, or continuous independent variables (confounders). We report the results with independent variables in model (4) being dichotomous, or dichotomous and continuous variables. Nevertheless, results were similar when categorical variables were included in model (4).

For simulations with only a dichotomous variable x₁, distributions of the dichotomous variable x₁ were either 50% = 0 and 50% = 1 or 66.6% = 0 and 33.3% = 1. For simulations with a dichotomous variable, x₁, and a continuous variables, x₂, when x₁ = 0, the mean of x₂ was either 1 or −1, and the variance was held constant at 1; when x₁ = 1, the mean of x₂ ranged from −4 to 4 by increments of 1, and the variance was held constant at 4. We also evaluated different values of the coefficients β_x1z, β_x2z, β_x1y, and β_x2y to reflect differing strengths of association with treatment and outcome. For dichotomous x₁, we evaluated positive and negative values of 0.69, 1.39, and 1.79, which correspond to odds ratios of 2, 3, and 4 when positive. For the continuous variable x₂, simulations used values of 0.3, 0.6, and 1.2 for β_x2z and β_x2y.

We then generated the dichotomous treatment variable z_i based on the treatment probability model (4), i = 1 to 500.

Probability model for the outcome, y. The dichotomous outcome variable, y, was simulated, based on the following model:

(5)

where α_y is the intercept and equals to 0.69, β_zy is the coefficient for the association between treatment and outcome and is assigned zero to assess the type I error rates. X are confounders and β_xy are the corresponding coefficients, and their values are the same as those of β_xz in (4). The dichotomous outcome variable y_i was generated based on the outcome probability model (5), i = 1 to 500.

Analysis of each simulated dataset. For each dataset we fit the propensity score models, obtained the IPTW and SWs, and then calculated the sample sizes in the pseudo data and fit outcome model. 5000 datasets were simulated and analyzed for each combination of parameters.

An Example

In a recent study examining the effectiveness of serum potassium monitoring in reducing hyperkalemia-associated adverse events during the first year of therapy, 27,355 diabetic patients newly prescribed with a renin-angiotensin-aldosteronesystem (RAAS) inhibitor between January 1, 2001 and December 31, 2006 were retrospectively identified. Table 3 shows that the patients with and without serum potassium monitoring in the original cohort were significantly different on many demographic and clinical characteristics. Nearly three-fourths of this cohort had serum potassium monitoring during their study follow-up period. This study is an example of when matching by propensity scores would not be optimal because the majority of those with serum potassium monitoring would be omitted because of a smaller number of those without serum potassium monitoring.

We fit a logistic regression model to obtain the propensity scores and included the following variables: use of digoxin, use of diuretic, use of potassium supplements, study site, sex, drug groups of RAAS inhibitor, age, kidney transplant, a drug-dispensing-based chronic disease score based on a modification of the method of Clark et al. [22], potassium monitoring within 6 months before study entry, diagnosis of hyperkalemia within 6 months before study entry, inpatient hospitalization or emergency department visit within 6 months before study entry, the presence of heart failure, and the presence of chronic kidney disease. The SW-adjusted results of characteristics comparisons are presented in Table 3 as well. All covariates except age, chronic disease score, and the use of potassium supplements became comparable after SW adjustment between those whose potassium was monitored and those whose potassium was not monitored (see Table 3). Although age, chronic disease score, and the use of potassium supplements remained statistically different between groups, the magnitudes of difference were markedly reduced.

The sample size in the pseudo data using the SWs was 27,407 compared with 54,891 using IPTW. The sample size in the pseudo data using the SWs was only slightly larger than the original 27,355, and the impact on variance estimate of treatment effect was minimal. The incidence rate ratio and confidence intervals were 0.46 (0.34, 0.61) using SWs compared with 0.46 (0.38, 0.55) using typical IPTW. While adjusting for age, the use of potassium supplements, and chronic disease score using SWs, the incidence rate ratio was 0.49 (0.37, 0.66), which was very close to the results without the adjustment of these covariates in outcome model, indicating that the balance of age, the use of potassium supplements, and chronic disease score between the two groups with SWs was sufficient. Comparison of these two weights from this example showed that IPTWs have larger standard deviations and wider ranges than SWs (Table 4).

Discussion

In this article, we demonstrate several advantages of SWs over IPTWs in analyzing data obtained from observational studies. First, using SWs can reduce the weights of either those treated subjects with low propensity scores or those untreated subjects with high propensity scores in the pseudo data sets. Our serum potassium monitoring example showed that IPTWs have larger standard deviation and wider range than SWs (Table 4). Thus, results using SWs are robust even with few observations with extreme IPTWs. Second, unlike variance estimators, no additional steps are needed when SWs are used because the SW approach provides appropriate variance estimates and confidence intervals of treatment effect from conventional regression models for fitting the outcome variables. Third, computer programming is simple for one to use SWs to obtain the effect of treatment effects and confidence intervals as compared with the bootstrap approach. One only needs to calculate the weights differently. Fourth, in our simulation studies and example, outcome variables are dichotomous. Unlike those developed estimators, the SW approach is applicable to outcome variables (e.g., dichotomous, continuous, and count data) that have a finite distribution. Our simulation results also show that SW is a reasonable alternative to the robust variance estimator and has the advantage of reducing influential weights.

The limitation of the SW approach is the uncertainty of the influence of continuous confounders when their association with the probability of being treated and outcome is very strong. As shown in simulation studies, the sample size in some of the pseudo data sets can be different from the original data set when the confounding effect is strong. Nevertheless, it is uncommon because our simulation results showed that the mean sample size approximated the original sample size. It is recommended that one always examine the difference between sample sizes in the original cohort and the pseudo cohort. When there is evidence that the sample size of the pseudo data is different from that of the original data set, one can use the robust variance estimator with IPTWs although this latter method can produce slightly larger standard errors.

Conclusion

Our theoretical, simulation results and the real data example demonstrate that the use of the SWs in the pseudo data preserves the sample size close to the original data. In addition, we conclude that use of SWs produces the appropriate estimation of the variance of the main effect and maintains an appropriate type I error rate. SWs may be a useful tool to balance confounders between groups in observational studies.

Source of financial support: Diabetes and Drug-Associated Hyperkalemia: Effect of Laboratory Monitoring is supported by the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) under the following grant number: R21DK075076.