Packet randomized experiments for eliminating classes of confounders

Glossary

Introduction

Two major methods for studying causality are ordinary association studies and randomized controlled experiments or trials (RCTs). Each has their advantages and disadvantages. The greatest disadvantage of association studies is that ‘association does not necessarily imply causation’ (to which we might add the less often acknowledged corollary that ‘a lack of association does not necessarily imply a lack of causation’). The reasons for this indeterminacy are also well known: the possible presence of confounding variables and the possibility that the cause–effect relationship could be reversed (i.e. that instead of X causing Y, Y causes X). RCTs overcome the problems of endogeneity and confounding by randomly assigning subjects to levels of the independent variable (e.g. X₁), thereby assuring that the population distribution of X₁ is independent of all known and unknown (measured and unmeasured) prerandomization variables that might otherwise be confounders 4. The greatest limitations of RCTs are that they are often impractical, prohibitively expensive, unethical or simply impossible because subjects cannot be randomly assigned to levels of X₁.

Majumdar and Soumerai 5 have argued that a false dichotomy exists in which RCTs and association studies are often treated as the only alternatives; consequently, researchers do not consider other study designs that occupy an intermediary inferential position between the pure RCT and the ordinary observational association study. The result is that decisions about clinical, public health and policy actions are founded on a less-than-optimal understanding of the relationship between causes and effects, interventions and outcomes.

Herein, we introduce general principles of an intermediary experimental design that is inferentially stronger than an ordinary association because it can rule out entire classes of both measured and unmeasured potential confounders, but is not as inferentially strong as a pure RCT, which can rule out all potential confounders. Although specific cases of this design have been used in the past, including by us 6, a systematic explication of its general form and inferential properties has, to our knowledge, never been published.

As such, we: (a) introduce the idea of packet randomized experiments (PREs) and their inferential properties; (b) give examples of the use of PREs from both published and planned studies (Table 1); (c) discuss genetic linkage analysis as a special case of a PRE; (d) explain how, in some PREs, conditioning on particular covariates can indirectly and asymptotically control for some of the confounders left uncontrolled by the design; and (e) end with a general discussion.

Introduction to PREs and their inferential properties

Let S_i (for i = 1 to N) denote the subjects (units of observation) in a study, with N being the number of subjects. In biomedical and behavioural research, subjects most often will be individual humans or individual model organisms (e.g. mice) but in principle could be clusters of individuals (e.g. classrooms, cities, mouse litters and ant colonies) or inanimate objects (e.g. buildings, airplanes, etc.). Let Y be some outcome variable of interest measured on S₁ to S_N. Y could be continuous (e.g. height, BMI and IQ), dichotomous (e.g. dead or not dead), ordinal (e.g. number of sales made) or time to event (e.g. survival time). Let X₁ denote some independent variable of interest – some variable hypothesized to have a causal effect on Y. X₁ could be a characteristic of a blood sample with which a subject is transfused, a subject's college flatmate, a characteristic of a book one is assigned to read, a characteristic of a car one has rented, a dietary constituent of a fruit one is assigned to eat, and so on. Suppose that, for some reason, we are not willing or able to randomly assign S's to levels of X₁ and only X₁. However, we can randomly assign S's to objects or situations that differ in their values of X₁, as well as in their values of other variables, X₂ to X_K, with the identities of X₂ to X_K not necessarily being known and, indeed, the value of K being unknown in most cases. We will refer to the objects or situations to which the S's can be assigned as ‘packets’ (P), because they are packets of the variables X₁ to X_K. For example, the random assignment of a subject to a flatmate constitutes a treatment that includes the myriad of characteristics of the assigned flatmate. Although our independent variable of interest may be the academic performance of the flatmate, we must also contend with the cleanliness, social habits or any other characteristic of the flatmate that may impact the outcome of interest.

One can easily imagine experiments in which each S was assigned to more than one P or more than one S was assigned to each P (see our example on altitude in section 4), but, for simplicity, for the remainder of this section, we will assume that each S is assigned to one and only one P. Hence, let P_i (for i = 1 to N) denote the packets in the experiment. Following our examples of potential variables for X₁ above, packets could be blood samples with which subjects are transfused, subjects’ college flatmates, books subjects are assigned to read, cars subjects are assigned to rent, fruits subjects are assigned to eat, and so on. Let W₁ to W_c denote characteristics of S that, by definition, are not within the set of X₂ to X_k. Examples of W's could include intrinsic subject characteristics (e.g. genotypes, parental occupation, personality traits, wealth, tastes, preferences and behaviours) and contextual factors. When randomization of S's to X's is not used, the subject characteristics W₁…_C may be correlated with the levels of X₁ to which S's were exposed (e.g. a correlation between academic achievement of the flatmate (X₁) and prior valuation of academic achievement by the subject (W). If W₁…_c are observed and measured, they can be statistically controlled. If they are unobserved, as they often are, they may confound the relationship between X₁ and the outcome of interest. In contrast, when S's are randomized to X's, the random assignment helps eliminates the possibility of confounding due to observed and unobserved W₁…_C variables. This is the key strength of PREs and distinguishes a PRE from an observational study. It does not, however, eliminate potential correlations between the treatment of interest X₁ and other packet characteristics X₂…_k. This is the key limitation of PREs and distinguishes a PRE from a standard randomized controlled trial.

Suppose we were to study S-P pairings created by means other than random assignment (e.g. by allowing S's to choose packets with which to be paired) and fit a model of the form f(Y_i) = g(X_1,j) + e_i where f(Y) denotes some link function of Y, g(X₁) denotes some function of X₁, and e_i denotes an error term 7. Could we then assume that the fitted model represents a model of causation? No. This is because the true model might be f(y_i) = g(X_1,i) + h(X₂,…X_K) + m(W₁,…W_c) +e_i, where h and m are unspecified functions of packet and subject characteristics, respectively. Under such a model, if X₂ to X_K and W₁ to W_C are not independent of X₁, then the observed relation between X₁ and Y will be confounded and may not represent a causal effect. However, if we randomly assign S's to P's, then X₁ must be independent of W₁ to W_C, and therefore, W₁ to W_C cannot be confounders of any relation between X₁ and Y. This follows from the same logic of randomization that underlies the use of standard randomized controlled trials (RCTs) to eliminate confounders. PREs and standard RCTs do not differ in their use of randomization to eliminate the possibility of confounding due to subject variables; rather, they differ only in whether subjects are assigned to a single treatment or to a ‘packet’ of treatments. The degree to which PREs improve causal inferences via randomization is directly related to the proportion of any confounding that occurs which is due to W₁…_C. As the proportion of confounding due to W₁…_C approaches 1, so too do PREs approach the inferential strength (ability to rule out causation) possessed by standard RCTs. Unfortunately, the proportion of confounding due to W₁…_C vs. X₁…_k will usually be unknowable, thus one of the key limitations of PREs is that the exact degree of improvement in causal inferences will usually be unknown and variable among studies. Nonetheless, because PREs eliminate potential confounders (W₁ to W_C), they strengthen causal inferences relative to standard observational studies of associations between X₁ and Y. An additional limitation of PREs is that they cannot rule out potential confounding from X₂ to X_K. As will be discussed later, sometimes such potential confounding can be mitigated using the more familiar methods of statistical adjustment in association studies, and in some situations, this potential confounding actually has utility. PREs have been used to improve causal inferences in a variety of disciplines, as discussed in the following section.

Examples of PREs

Human milk to mice

Significant ethical considerations in designing breastfeeding RCTs make it difficult to study the purported benefits of breastfeeding exposure while disentangling both genetic and environmental factors that influence (both directly and indirectly vis-à-vis maternal contact) breastfeeding behaviour and breast milk composition. Wide-ranging benefits of breastfeeding, including a protective effect against obesity, have been touted by public health and professional organizations throughout the world based primarily on observational evidence. The observed benefits, however, are diminished when known modulators are controlled (i.e. infant exposure to smoking, socioeconomic status, etc.) 13-15. Our conceptualized PRE specifically related to breast feeding studies offers a method to control for a significant number of modulators that breastfeeding studies to date have not, and ethically cannot, control for, namely e_h (uncontrolled human external confounders with direct effects on the infant), τ_h (baby phenotype attributed to maternal factors), α_h human maternal factors directly impacting the composition of breast milk); thus, the criteria to make causal claims about the effect of breast milk on obesity is not met (Fig. 1).

Our group (Fields and Allison) is proposing a breastfeeding PRE that eliminates ethical constraints while improving statistical inference that most studies cannot achieve by introducing a study design (Fig. 1) where mice will be randomized to human breast milk obtained from breastfeeding mothers who vary greatly in a diverse range of physiological (fatness), sociological (socioeconomic status, education, urban vs. rural) and genetic confounders that human correlational studies could not remove. By randomizing mice (subjects) to differing sets of human breast milk (packets), the PRE design eliminates τ_m (mouse phenotype attributable to mouse maternal factors given all mice have the same mother), α_h (impact of human maternal factors on the composition of breast milk), and e_m (external confounders directly affecting the mouse phenotype, given all mice are housed and raised in the same tightly regulated/controlled environment). All three sources (τ_m, α_h and e_m) of confounding cannot be eliminated in human studies; however, the proposed PRE allows for greater inferential power than associational inferences currently being conducting in human studies by eliminating all classes of confounders while retaining physiological relevance in the proposed animal study.

Genetic linkage and family-based association studies as special cases of PREs

In genetic studies, the potential sources of confounding are numerous, encompassing the entire genome. Within any genetic population – that is, a group of individuals who could theoretically mate randomly – actual matings are natural experiments that can be taken advantage of analytically. Two types of analysis do this, linkage analysis and family-based association testing. In both cases, the genomes of the two parents are the source of all possible genomes of any offspring. From a theoretical standpoint, we can consider the nuclear family to have been ‘randomized’ to the set of genomes determined by the parental genomes. These analyses are thus special cases of packet randomization in which the packet is identified with the nuclear family.

The practical difference between these types of studies and previously given examples of packet randomization is a matter of perspective. Previous examples considered the definition of a packet to be the primary variable of interest. In the cases of these types of genetic analysis, the primary variable of interest can vary over the genome. Linkage analysis seeks evidence that the inheritance of a particular chromosomal segment from a parent will be associated with some observable trait. Family-based association analysis seeks evidence that the inheritance of a specific allele at a specific locus from a parent will be associated with some observable trait. Central to both analyses is the packet: because all genomes in all children have to be derived from just the two genomes of the parents, there is always a probability of ½ that a particular allele or chromosomal segment came from the same parental source in two siblings. The analysis is thus performed on the background of this average sharing, giving us a powerful way of controlling for confounding on an immense scale.

Linkage analysis attempts to determine whether the degree of local genetic sharing at a specific position on a chromosome is correlated with the observed concordance of the trait of interest. We can consider this to be a question of the form, ‘All other (genetic) factors being (roughly) equal, does concordance of inheritance at one position correlate with concordance of some observed trait’? The degree of correlation of the trait is decomposed into a portion that is proportional to the overall sharing of the entire genome and a portion that is proportional to the degree of genetic sharing at a specific position in the genome. Thus, unlike previously discussed examples of packet randomization, the primary variable of interest in linkage analysis (the inherited chromosomal segment, which can include multiple variants that all influence the trait) believed to be associated with a particular phenotype is itself unobserved and must be inferred through statistical analysis.

Conditioning on particular covariates to control for packet confounders

The major limitation of PREs is the inability to eliminate confounders at the level of the packet. PREs are not alone in having this limitation – any nonrandomized analysis in which dependent variables are a function of independent variables is at risk of confounding due to unobserved independent variables not included in the model. In this section, we propose a method in which controlling for certain packet-level covariates can reduce the risk of confounding due to omitted variables.

Section 5 described a PRE in which service members were quasi-randomly assigned to a city, and change in weight was observed. The benefit of this research design is the elimination of endogeneity and subject factors as threats to the observed changes in weight. However, causal inferences about the effect of the new city's characteristics on weight can be made only for the city as whole, not for a specific characteristic of the city. Suppose that the effect of altitude (A) on weight change (Y) among subjects is of interest. Standard PREs are unable to distinguish between the effect of altitude and the myriad other city level characteristics that may affect weight. Temperature, sunlight exposure and cultural norms all vary across cities and plausibly confound the relationship between altitude and weight change. Even if we were to include all known or measured characteristics of a city, it is unlikely that we have accounted for all variables that vary across cities, are associated with altitude, and have an effect on weight.

Nonetheless, it may be possible to identify a variable that plausibly serves as a vector of all unobserved city level characteristics and can asymptotically control for all packet-level characteristics left uncontrolled by the design. In the case of investigating the effects of city characteristics on weight, we propose that mean weight of a city (Z) is such a variable (See Fig. 2). Mean weight of a city necessarily represents the average effect of all city characteristics on the weight of all individuals living within that city. This is true regardless of whether some city characteristics have an effect on the weight of only select individuals, or whether some residents have lived in the city longer than others. For example, some individuals may live in an area of a city more conducive to physical activity than others. Although these subjects are exposed to an array of city characteristics that other subjects living in the same city are not, the uneven distribution of city characteristics and their effects on weight will be reflected in average city weight. The same is true for length of residence – although some city residents will have been exposed to that city's environment for only a few weeks or perhaps even a few days, other city residents will have lived in the city their whole lives and therefore have been exposed to whatever aspects of the city influence weight. The uneven exposure to city characteristics due to varying lengths of residence will be reflected in the average city weight. The average city effect on weight also includes all individual-level effects on weight in that city, including average city socioeconomic status, demographics or any other aggregate of individual-level characteristics. One important aspect of using average city weight is that it is the average of the outcome variable and, as such, it equals the city level effect plus the average of the city level random variation, plus the average of independent level effects. Thus, it is the city level effect plus a term that is O(1/N), where N is the number of observed individuals. Finally, although distinctions between individual and neighbourhood-level characteristics are essential in associational studies of neighbourhood characteristics, such a distinction is less important in PREs. From the perspective of the randomized subjects, aggregate individual effects are simply another type of city characteristic, while the random assignment of subjects to cities assures that between-subject variation in weight is not due to omitted subject-level variables.

Figure 3 presents the model-implied variance–covariance matrix for the model in Fig. 1, in which average weight is assumed to be a vector of all observed and unobserved city characteristics that affect weight. Unless otherwise defined, the symbols used in the following equation come from the variance–covariance expression in Fig. 3. If we naively assume that altitude is correlated with weight change and uncorrelated with any other factor, the estimated regression coefficient of altitude on weight change (β_Y) is equal to the observed covariance between altitude and weight change over the variance in altitude – the standard regression coefficient formula 26:

However, if altitude is correlated with other variables (W), and these variables are correlated with weight, failure to include them in the model results in an estimated effect of altitude that is biased by the correlation between altitude and unobserved city characteristics (ρ). Using covariance algebra, it can be shown that failure to model the correlation between altitude and unobserved characteristics biases the estimated covariance between Y and A(β_Y) to the extent that it incorrectly includes the covariance between Y, A and all other variables not included in the model:

If we regress Y on A and Z, writing the regression as Y = δ_AA + δ_ZZ , the true values of the coefficients are given by:

If W were observable and could be included in a regression, the ‘true’ coefficient of A would be β_Y. The question of interest is the relationship between β_Y and δ_A – the coefficient that would result whether we included only observable quantities and failed to adjust for all unobserved variables. If Z is close to being a good surrogate for W, the correlation between them is close to one, and the term (φ_Wφ_A - ρ²) will be close to zero. As (φ_W)φ_A - ρ² approaches zero, the regression coefficients will approach:

The first term is precisely the slope of a regression line containing only the single predictor A. Thus, if the observed variable is a perfect surrogate for the unobserved ‘true’ predictor, the other term will simply drop out of the model. The other limiting case would be if the two variables are unrelated so that ρ would be close to zero. In this limit, the coefficients would be:

Conditioning on a particular covariate may thus help to adjust for packet-level factors not accounted for via randomization. Specifically, it may be possible to condition on a variable that represents the average of the outcome of interest over individuals who are within the packet but who are not themselves included in the experiment 27.

Discussion

This article had three goals. First, we sought to provide a general introduction to packet randomization, an experimental design, as an intermediary between observational studies and true randomized trials. The false dichotomy between association studies and the pure RCT obscures the potential of natural experiments to improve causal claims 5. Failure to consider intermediate designs between observational studies and true randomized trials limits the quality of evidence available to policymakers. Although the internal validity of packet randomized experiments cannot match that of randomized trials, it is a step above typical observational studies. Most notably, PREs greatly improve upon the strength of causal inferences by eliminating subject-level characteristics as potential confounders. In doing so, they compare favourably with other methods to control for confounding in nonexperimental research, including instrumental variable methods and propensity score matching.

Instrumental variable methods attempt to control for confounding via the identification of a variable (Z) whose effect on the outcome variable (Y) operates only through the independent variable of interest (X). Instrumental variable methods are an attempt to isolate and use the variation in X not correlated with the errors (U) 28. If the instrument is truly an exogenous predictor (uncorrelated with the error terms, U) of X, estimates of its effects on Y will be consistent (bias will reduce to zero as sample size increases 29). A limitation of instrumental variable methods is that it is not possible to empirically determine whether Z is exogenous to U; rather, this must be determined through judgment 30. In PREs, randomization rather than judgment is used to assure the treatment X is exogenous to U for the portion of covariance between X and Y due to subject factors W₁…W_c. PREs cannot, however, assure that X is exogenous to U for the portion of covariance between X and Y due to unmeasured packet characteristics X₂…X_k. Additional methods, including instrumental variable approaches, may be considered to potentially reduce confounding from packet characteristics.

Another method frequently used to improve causal inferences in observational studies is propensity score analysis. Propensity scores indicate the estimated probability that an individual has a particular level of the independent variable of interest conditional on other characteristics of that individual 30. Propensity scores can then be used reduce bias in the estimated independent variable effect by conditioning on differences among S's in propensity. This method is, in effect, an attempt to use a single scalar variable (the propensity score) to match participants as closely as possible on all covariates and who differ only on whether they received the treatment. The model used to match participants and estimate a propensity score can include a large number of covariates and interactions; allowing the analyst to estimate a simpler final model 31. Propensity score methods and more standard multiple regression methods will produce the same estimates of a treatment effect so long as the model used to estimate the propensity score is used in the multiple regression analysis 31. Despite the advantages of reducing bias in the estimated treatment effect using a single scalar variable, propensity score matching still depends on the inclusion of all relevant variables, measuring them without error, and modelling their functional form correctly to provide unbiased effect estimates. If relevant variables are unmeasured (or poorly measured) or unknown, confounding can remain despite use of propensity scores. In contrast, PREs use randomization to ensure that S's characteristics do not systematically differ in association with levels of the independent variable.

The second goal of this work was to describe the application of PREs across a range of disciplines and to demonstrate their potential to investigate any outcome within almost any discipline. PREs have been applied in animal, genetic and social scientific research. The lack of a general introduction to PREs makes it more likely that researchers within specific disciplines must spend unnecessary time reinventing for themselves a method that already has been employed and developed by researchers in multiple disciplines. With this comes the risk that insights of past researchers will be overlooked, or that old mistakes will be repeated. Relatedly, we hope that greater awareness of the PREs among researchers across all disciplines, especially those in human behavioural disciplines, will encourage the search for opportunities whereby PREs can be used to improve causal claims previously made using only observational data.

Third, this work sought to describe a situation in which conditioning on a covariate can plausibly control for packet-level confounders. Although such an approach cannot address packet-level confounding variables as convincingly as randomization of subjects to packets, it enables researchers to better isolate the effect of a single packet characteristic from the overall effect of the set of packet characteristics. The inability to isolate a single causal factor within a complex system is typically viewed as a limitation; however, we believe that, in some situations, it may be a strength. The interactions of variables within highly complex systems such as a brain or a city initially may require consideration of their effects as a whole rather than understanding the effect of any particular factor in isolation. Multifactorial outcomes such as obesity provide an excellent example of the utility in assessing a packet of exposures simultaneously. If the entire packet of exposures is beneficial, from a public health perspective, it may not be immediately necessary to isolate the specific exposure within the packet that is most helpful. Rather, the entire packet might be applied.

Although PREs combine many of the strengths of associational and true RCT studies, they are not a panacea. Researchers must continue to pay attention to the assumption of randomization, generalize causal claims only to the population of subjects for whom randomization actually occurred and consider the possibility of ‘supra-packet’ factors such as time – any factors not accounted for by randomization. The assumption of randomization is most important, and those who wish to use PREs in their own analyses may wish to follow the example of Anderson et al. 8, who provided both deductive and empirical evidence to support their assumption that randomization of cases to judges occurred. If evidence for randomization cannot be provided, PREs cannot be claimed to generate causal inferences superior to that of observational studies. Researchers must also be careful about generalizing claims (about either causation or association) beyond the population of subjects studied with a PRE, lest they introduce another confound into their analysis – whether or not subjects were randomized. Students who opt not to participate in the random assignment of flatmates, districts in which case randomization was more plausible than others, and service members randomized to a city to live, may, likely do, systematically vary from the students, districts and nonservice members where randomization did not occur. The usefulness of PREs, as with standard RCTS, thus depends on whether the assumption of randomization is correct for the population of interest, or whether those randomized resemble the population of interest. If randomization is plausible for the population of interest, and supra-packet factors such as time are accounted for, well-designed PRE's can offer inferentially superior and more useful results than association studies.

Disclosures

None.

Grant support

Research reported in this publication was supported by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health under Award Numbers T32DK062710, P30DK056336 and R25DK099080. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Diabetes and Digestive and Kidney Diseases or the National Institutes of Health.

Address

Office of Energetics, Nutrition Obesity Research Center, University of Alabama at Birmingham, LHL 407, 1700 University Boulevard, Birmingham, AL 35294, USA (G. Pavela); Department of Epidemiology, University of Alabama at Birmingham, Ryals Building, Room 217D, 1665 University Boulevard, Birmingham, AL 35294, USA (H. Wiener); Department of Health Behavior, University of Alabama at Birmingham, Ryals Building, Room 241C, 1665 University Boulevard, Birmingham, AL 35294, USA (K. R. Fontaine); Department of Pediatrics, University of Oklahoma Health Sciences, 1200 Children's Avenue Suite 4500, Oklahoma City, OK 73104, USA (D. A. Fields); Epidemiology Consult Division, U.S. Air Force School of Aerospace Medicine, 2510 5th Street, Bldg 840, Wright Patterson Air Force Base, OH 45433, USA (J. D. Voss); Office of Energetics, Nutrition Obesity Research Center, University of Alabama at Birmingham, Ryals Building, Room 140J, 1665 University Boulevard, Birmingham, AL 35294, USA (D. B. Allison).