Analytic posteriors for Pearson's correlation coefficient

1 Introduction

Pearson's product–moment correlation coefficient ρ is a measure of the linear dependency between two random variables. Its sampled version, commonly denoted by r, has been well studied by the founders of modern statistics such as Galton, Pearson, and Fisher. Based on geometrical insights, FISHER (1915, 1921) was able to derive the exact sampling distribution of r and established that this sampling distribution converges to a normal distribution as the sample size increases. Fisher's study of the correlation has led to the discovery of variance-stabilizing transformations, sufficiency (FISHER, 1920), and, arguably, the maximum likelihood estimator (FISHER, 1922; STIGLER, 2007). Similar efforts were made in Bayesian statistics, which focus on inferring the unknown ρ from the data that were actually observed. This type of analysis requires the statistician to (i) choose a prior on the parameters, thus, also on ρ, and to (ii) calculate the posterior. Here we derive analytic posteriors for ρ given a large class of priors that include the recommendations of JEFFREYS (1961), LINDLEY (1965), BAYARRI (1981), and, more recently, BERGER and SUN (2008) and BERGER et al. (2015). Jeffreys's work on the correlation coefficient can also be found in the second edition of his book (JEFFREYS, 1961), originally published in 1948; see ROBERT et al. (2009) for a modern re-read of Jeffreys's work. An earlier attempt at a Bayesian analysis of the correlation coefficient can be found in JEFFREYS (1935). Before presenting the results, we first discuss some notations and recall the likelihood for the problem at hand.

2 Notation and result

Let (X₁,X₂)^′ have a bivariate normal distribution with mean and covariance matrix

where and are the population variances of X₁ and X₂, and where ρ is

(1)

Pearson's correlation coefficient ρ measures the linear association between X₁ and X₂. In brief, the model is parametrized by the five unknowns θ=(μ₁,μ₂,σ₁,σ₂,ρ).

Bivariate normal data consisting of n pairs of observations can be sufficiently summarized as , where

is the sample correlation coefficient, the sample mean, and the average sums of squares. The bivariate normal model implies that the observations y are functionally related to the parameters by the following likelihood function:

(2)

For inference we use the following class of priors:

(3)

where η denotes the hyperparameters, that is, η=(α,β,γ,δ). This class of priors is inspired by the one that José Bernardo (2015) used in his talk on reference priors for the bivariate normal distribution at the ‘11th International Workshop on Objective Bayes Methodology in honor of Susie Bayarri’. This class of priors contains certain recommended priors as special cases.

If we set α=1,β=γ=δ=0 in Equation 3, we retrieve the prior that Jeffreys recommended for both estimation and testing (JEFFREYS, 1961, pp. 174–179 and 289–292). This recommendation is not the prior derived from Jeffreys's rule based on the Fisher information (e.g., LYet al., 2017), as discussed in BERGER and SUN (2008). With α=1,β=γ=δ=0, thus, a uniform prior on ρ, Jeffreys showed that the marginal posterior for ρ is approximately proportional to h_a(n,r|ρ), where

represents the ρ-dependent part of the likelihood Equation 2 with θ₀=(μ₁,μ₂,σ₁,σ₂) integrated out. For n large enough, the function h_a is a good approximation to the true reduced likelihood h_γ,δ given below. †

If we set α=β=γ=δ=0 in Equation 3, we retrieve Lindley's reference prior for ρ. LINDLEY (1965, pp. 214–221) established that the posterior of is asymptotically normal with mean and variance n⁻¹, which relates the Bayesian method of inference for ρ to that of Fisher. In Lindley's (1965, p. 216) derivation, it is explicitly stated that the likelihood with θ₀ integrated out cannot be expressed in terms of elementary functions. In his analysis, Lindley approximates the true reduced likelihood h_γ,δ with the same h_a that Jeffreys used before. BAYARRI (1981) furthermore showed that with the choice γ=δ=0, the marginalization paradox (DAWID et al., 1973) is avoided.

In their overview, BERGER and SUN (2008) showed that for certain a,b with α=b/2−1,β=0,γ=a−2, and δ=b−1, the priors in Equation 3 correspond to a subclass of the generalized Wishart distribution. Furthermore, a right-Haar prior (e.g., SUN AND BERGER, 2007) is retrieved when we set α=β=0,γ=−1,δ=1 in Equation 3. This right-Haar prior then has a posterior that can be constructed through simulations, that is, by simulating from a standard normal distribution and two chi-squared distributions (BERGER AND SUN, 2008, Table 1). This constructive posterior also corresponds to the fiducial distribution for ρ (e.g., FRASER, 1961; HANNIGet al., 2006). Another interesting case is given by α=0,β=1,γ=δ=0, which corresponds to the one-at-a-time reference prior for σ₁ and σ₂; see also Jeffreys (1961 p. 187).

The analytic posteriors for ρ follow directly from exact knowledge of the reduced likelihood h_γ,δ(n,r|ρ), rather than its approximation used in previous work. We give full details, because we did not encounter this derivation in earlier work.

Theorem 1.The reduced likelihood hγ,δ(n,r|ρ) If |r|<1,n>γ+1 , and n>δ+1 , then the likelihood f(y|θ) times the prior Equation 3 with θ₀=(μ₁,μ₂,σ₁,σ₂) integrated out is a function f_γ,δ that factors as

(4)

The first factor is the marginal likelihood with ρ fixed at zero, which does not depend on r nor on ρ , that is,

(5)

where . We refer to the second factor as the reduced likelihood, a function of ρ which is given by a sum of an even function and an odd function, that is, h_γ,δ=A_γ,δ+B_γ,δ , where

(6)

(7)

where and where ₂F₁ denotes Gauss' hypergeometric function.

This main theorem confirms Lindley's insights; h_γ,δ(n,r|ρ) is indeed not expressible in terms of elementary functions, and the prior on ρ is updated by the data only through its sampled version r and the sample size n. As a result, the marginal likelihood for data y then factors into p_η(y)=p_γ,δ(y₀)p_α,β(n,r;γ,δ), where is the normalizing constant of the marginal posterior of ρ. More importantly, the fact that the reduced likelihood is the sum of an even function and an odd function allows us to fully characterize the posterior distribution of ρ for the priors Equation 3 in terms of its moments. These moments are easily computed, as the prior π_α,β(ρ) itself is symmetric around zero. Furthermore, the prior π_α,β(ρ) can be normalized as

(13)

where denotes the beta function. The case with β=0 is also known as the (symmetric) stretched beta distribution on (−1,1) and leads to Lindley's reference prior when we ignore the normalization constant, that is, , and, subsequently, let .

Corollary 1.Characterization of the marginal posteriors of ρ If n>γ+δ−2α+1 , then the main theorem implies that the marginal likelihood with all the parameters integrated out factors as p_η(y)=p_γ,δ(y₀)p_α,β(n,r;γ,δ) where

(14)

defines the normalizing constant of the marginal posterior for ρ . Observe that the integral involving B_γ,δ is zero, because B_γ,δ is odd on (−1,1) . More generally, the k th posterior moment of ρ is

(15)

These posterior moments define the series

(16)

where is the normalization constant of the prior Equation 13, W_γ,δ(n) is the ratios of gamma functions as defined under Equation 7, and refers to the Pochhammer symbol for rising factorials. The terms a_k,m and b_k,m are

The series defined in Equation 16 are hypergeometric when β is a non-negative integer.

3 Analytic posteriors for the case β=0

For most of the priors discussed earlier, we have β=0, which leads to the following simplification of the posterior.

Corollary 1. (Characterization of the marginal posteriors of ρ, when β=0)If n>γ+δ−2α+1 and |r|<1 , then the marginal posterior for ρ is

(17)

where p_α(n,r;γ,δ) refers to the normalizing constant of the (marginal) posterior of ρ , which is given by

More generally, when β=0 , the k th posterior moment is

when k is even, and

when k is odd.

The marginal posterior for ρ updated from the generalized Wishart prior, the right-Haar prior, and Jeffreys's recommendation then follow from a direct substitution of the values for α,γ, and δ as discussed under Equation 3. Lindley's reference posterior for ρ is given by

which follows from Equation 17 by setting γ=δ=0 and, subsequently, letting .

Lastly, for those who wish to sample from the posterior distribution, we suggest the use of an independence-chain Metropolis algorithm (TIERNEY, 1994) using Lindley's normal approximation of the posterior of as the proposal. This method could be used when Pearson's correlation is embedded within a hierarchical model, as the posterior for ρ will then be a full conditional distribution within a Gibbs sampler. For α=1,β=γ=δ=0,n=10 observations and r=0.6, the acceptance rate of the independence-chain Metropolis algorithm was already well above 75%, suggesting a fast convergence of the Markov chain. For n larger, the acceptance rate further increases. The R code for the independence-chain Metropolis algorithm can be found on the first author's home page. In addition, this analysis is also implemented in the open-source software package JASP (https://jasp-stats.org/).