4  Gaussian random vectors

Updated

August 4, 2026

This chapter introduces covariance as a description of second-order dependence and then develops multivariate Gaussian random vectors. Together, these ideas provide the basic language for linear Gaussian models.

4.1 Covariance of random vectors

We start with the definition of covariance of two real-valued random variables. We will then generalize the notion to random vectors.

4.1.1 Real-valued random variables

Let \(X\) and \(Y\) be real-valued random variables defined on the same probability space.

  • Covariance measures how two random variables vary together. In particular, let \(X\) and \(Y\) be jointly distributed random variables and let \(μ_X\) and \(μ_Y\) denote their means. Then, \[ \COV(X,Y) = \EXP[(X - μ_X) (Y - μ_Y)].\] Properties of expectation imply that \[ \COV(X,Y) = \EXP[XY] - \EXP[X] \EXP[Y]. \]

  • If \(\VAR(X)>0\) and \(\VAR(Y)>0\), the correlation coefficient between \(X\) and \(Y\) is defined as \[ρ_{XY} = \frac{\COV(X,Y)}{\sqrt{\VAR(X) \VAR(Y)}}.\]

  • The correlation coefficient satisfies \(\ABS{ρ_{XY}} \le 1\), with equality if and only if \[ \PR(Y = aX+b)=1 \] for some \(a \neq 0\) and \(b \in \reals\). [The proof follows from the Cauchy–Schwarz inequality, which we will study later.]

  • \(X\) and \(Y\) are said to be uncorrelated if \[\COV(X,Y)=0,\] or, equivalently, if \(\EXP[XY] = \EXP[X] \EXP[Y]\). This definition also applies when one of the random variables is constant.

  • Note that \[\begin{align*} \VAR(X + Y) &= \EXP[ ((X - \EXP[X]) + (Y - \EXP[Y]) )^2 ] \\ &= \VAR(X) + \VAR(Y) + 2\COV(X,Y). \end{align*}\] Thus, when \(X\) and \(Y\) are uncorrelated, we have \[ \VAR(X + Y) = \VAR(X) + \VAR(Y). \]

  • Independent random variables are uncorrelated but the reverse is not true.

Example 4.1 Consider the probability space \((Ω, \ALPHABET F, \PR)\) where \(Ω = [0, 2 π)\), \(\ALPHABET F\) is the Borel \(σ\)-algebra on \([0, 2 π)\) and \(\PR\) is the uniform distribution on \(Ω\). Define \(X(ω) = \cos ω\) and \(Y(ω) = \sin ω\). Show that \(X\) and \(Y\) are uncorrelated but not independent.

NoteSolution

To show that \(X\) and \(Y\) are not independent, consider the events \[ A = \{X > \tfrac12\} \quad\text{and}\quad B = \{Y > \tfrac12\}. \] The event \(A\) corresponds to \[ ω \in (0,\tfrac{π}{3}) \cup (\tfrac{5π}{3},2π), \] while \(B\) corresponds to \(ω \in (\tfrac{π}{6},\tfrac{5π}{6})\). Therefore, \[ \PR(A)=\PR(B)=\frac13, \qquad \PR(A\cap B)=\frac1{12}. \] Since \(\PR(A\cap B) \neq \PR(A)\PR(B)=\frac19\), the random variables \(X\) and \(Y\) are not independent.

Observe that

  • \(\displaystyle \EXP[X] = \int_{0}^{2 π} \cos ω \frac{1}{2 π}\, d ω = 0\).

  • \(\displaystyle \EXP[Y] = \int_{0}^{2 π} \sin ω \frac{1}{2 π}\, d ω = 0\).

  • \(\displaystyle \EXP[XY] = \int_{0}^{2 π} \cos ω \sin ω \frac{1}{2 π}\, d ω = \frac{1}{4π} \int_0^{2 π} \sin 2 ω\, d ω = 0\).

Thus, \[\EXP[XY] = \EXP[X]\EXP[Y].\]

4.1.2 Random vectors

The idea of covariance generalizes to random vectors. First we recall the definition of expectation for random vectors and random matrices.

  • For a random vector \(X = [X_1, \dots, X_n]^\TRANS \in \reals^n\), we have \[ \EXP[X] = \MATRIX{ \EXP[X_1] \\ \vdots \\ \EXP[X_n] }. \]

  • For a random matrix \(X = \MATRIX{ X_{1,1} & \cdots & X_{1,n} \\ X_{2,1} & \cdots & X_{2,n} \\ \vdots & \vdots & \vdots \\ X_{m,1} & \cdots & X_{m,n} } \in \reals^{m \times n}\), we have \[ \EXP[X] = \MATRIX{ \EXP[X_{1,1}] & \cdots & \EXP[X_{1,n}] \\ \EXP[X_{2,1}] & \cdots & \EXP[X_{2,n}] \\ \vdots & \vdots & \vdots \\ \EXP[X_{m,1}] & \cdots & \EXP[X_{m,n}] }. \]

With the above notation, we have the following.

  1. The covariance matrix of a random vector \(X \in \reals^n\) is defined as \[Σ_X = \COV(X) = \EXP[ (X - μ_X) (X - μ_X)^\TRANS].\]

  2. When \(X = [X_1, \dots, X_n]^\TRANS\), the covariance can be written as \[ Σ_X = \MATRIX{ \VAR(X_1) & \COV(X_1, X_2) & \cdots & \COV(X_1, X_n) \\ \COV(X_2, X_1) & \VAR(X_2) & \cdots & \COV(X_2, X_n) \\ \vdots & \ddots & \ddots & \vdots \\ \COV(X_n, X_1) & \COV(X_n, X_2) & \cdots & \VAR(X_n) } \]

  3. It is easy to see that \(Σ_X = Σ_X^\TRANS\). Thus, the covariance matrix is symmetric.

  4. If the components \(X_1, \dots, X_n\) are independent, then \(Σ_X\) is a diagonal matrix.

  5. \(Σ_X\) is positive semidefinite, i.e., its eigenvalues are real and non-negative. To see this, observe that for any deterministic vector \(v \in \reals^n\), we have \[\begin{align*} v^\TRANS Σ_X v &= v^\TRANS \EXP[ (X - μ_X) (X - μ_X)^\TRANS ] v \\ &= \EXP[ v^\TRANS (X - μ_X) (X - μ_X)^\TRANS v ] \\ &= \EXP[ w^2 ] \ge 0. \end{align*}\] where \(w = (X - μ_X)^\TRANS v\) is a scalar random variable.

  6. The cross covariance matrix of random vectors \(X \in \reals^n\) and \(Y \in \reals^m\) is an \(n × m\) matrix given by \[ Σ_{XY} = \COV(X,Y) = \EXP[ (X - μ_X) (Y - μ_Y)^\TRANS ]. \]

  7. Two random vectors \(X\) and \(Y\) are called uncorrelated if \(Σ_{XY} = 0_{n \times m}\). Equivalently, \[ \EXP[XY^\TRANS] = \EXP[X] \EXP[Y]^\TRANS. \]

  8. Two random vectors \(X\) and \(Y\) are called orthogonal if \[\EXP[X Y^\TRANS] = 0_{n \times m}. \]

  9. Let \(Z\) be the random vector obtained by stacking \(X\) and \(Y\): \[ Z = \MATRIX{X \\ Y} \in \reals^{n+m}. \] Its mean and covariance have the block structure \[ μ_Z = \MATRIX{μ_X \\ μ_Y} \quad\text{and}\quad Σ_Z = \MATRIX{ Σ_X & Σ_{XY} \\ Σ_{YX} & Σ_Y }, \] where \[ Σ_{YX} = Σ_{XY}^\TRANS. \]

4.1.3 Affine transformation of random vectors

  1. Mean and covariance. Let \(X \in \reals^n\) have mean \(μ_X\) and covariance \(Σ_X\). If \[ Y=AX+b \in \reals^m, \] where \(A \in \reals^{m \times n}\) and \(b \in \reals^m\) are deterministic, then \[ μ_Y=Aμ_X+b \quad\text{and}\quad Σ_Y=AΣ_XA^\TRANS. \]

  2. Cross-covariance. Let \(X \in \reals^n\) and \(Y \in \reals^p\) have cross-covariance \(Σ_{XY}\). If \[ U=AX+b \in \reals^m \quad\text{and}\quad V=CY+d \in \reals^q, \] where \(A \in \reals^{m \times n}\), \(C \in \reals^{q \times p}\), \(b \in \reals^m\), and \(d \in \reals^q\) are deterministic, then \[ Σ_{UV}=\COV(U,V)=AΣ_{XY}C^\TRANS. \]

For the first result, the property of the mean follows immediately from the properties of expectation. For the covariance, \[\begin{align*} Σ_Y &= \EXP[(Y-μ_Y)(Y-μ_Y)^\TRANS] \\ &= \EXP[A(X-μ_X)(X-μ_X)^\TRANS A^\TRANS] \\ &= AΣ_XA^\TRANS. \end{align*}\]

For the second result, \(U-μ_U=A(X-μ_X)\) and \(V-μ_V=C(Y-μ_Y)\). Therefore, \[\begin{align*} Σ_{UV} &= \EXP[(U-μ_U)(V-μ_V)^\TRANS] \\ &= A\EXP[(X-μ_X)(Y-μ_Y)^\TRANS]C^\TRANS \\ &= AΣ_{XY}C^\TRANS. \end{align*}\]

4.2 Multivariate Gaussian random vectors

4.2.1 Definition

Definition 4.1 A random vector \(X \in \reals^n\) is called Gaussian if every linear combination of its components is a scalar Gaussian random variable. Thus, for every deterministic vector \(a \in \reals^n\), \(a^\TRANS X\) is Gaussian.

If \(μ=\EXP[X]\) and \(Σ=\COV(X)\), we write \[ X \sim \mathcal{N}(μ,Σ). \] In particular, \[ a^\TRANS X \sim \mathcal{N}(a^\TRANS μ,a^\TRANS Σa), \qquad a \in \reals^n. \] A Gaussian random variable with zero variance is interpreted as a constant almost surely.

Random vectors \(X \in \reals^n\) and \(Y \in \reals^m\) are called jointly Gaussian if the stacked vector is Gaussian, i.e., \[ \MATRIX{X \\ Y} \sim \mathcal{N}\left( \MATRIX{μ_X \\ μ_Y}, \MATRIX{ Σ_X & Σ_{XY} \\ Σ_{YX} & Σ_Y } \right). \]

The mean vector and covariance matrix completely determine the distribution of a Gaussian random vector. This fact is what makes Gaussian models especially convenient: means and covariance matrices can replace full joint CDFs or PDFs.

ImportantGaussian marginals do not imply joint Gaussianity

It is not enough for each component to be Gaussian. Let \(X \sim \mathcal{N}(0,1)\), and let \(S\) be independent of \(X\) with \[ \PR(S=1)=\PR(S=-1)=\frac12. \] Define \(Y=SX\). Both \(X\) and \(Y\) have the \(\mathcal{N}(0,1)\) distribution, but \[ \PR(X+Y=0)=\frac12. \] Thus, \(X+Y\) is not a continuous random variable. It is also not constant, so it cannot be a Gaussian random variable (either nondegenerate or degenerate). Consequently, \((X,Y)\) is not jointly Gaussian.

4.2.2 Construction, density, and singularity

Let \(Z \sim \mathcal{N}(0,I_r)\), let \(L \in \reals^{n \times r}\) be deterministic, and define \[ X=μ+LZ. \] Then \[ X \sim \mathcal{N}(μ,LL^\TRANS). \] Conversely, if \(X \sim \mathcal{N}(μ,Σ)\) and \(Σ\) has rank \(r\), then \(Σ\) can be factored as \(Σ=LL^\TRANS\) and \(X\) has the same distribution as \(μ+LZ\). Thus, a valid Gaussian covariance matrix is always positive semidefinite.

  • If \(Σ\) is positive definite, then \(r=n\) and \(X\) has the PDF \[ f_X(x) =\frac{1}{\sqrt{(2π)^n\det(Σ)}} \exp\left(-\frac12(x-μ)^\TRANS Σ^{-1}(x-μ)\right), \qquad x \in \reals^n. \]

  • If \(Σ\) is singular, then \(X\) is supported on the lower-dimensional affine subspace \[ μ+\operatorname{range}(L) \] and does not have an ordinary PDF on \(\reals^n\).

4.2.3 Properties of Gaussian vectors

  1. Affine transformations. If \(X \sim \mathcal{N}(μ_X,Σ_X)\), \(A \in \reals^{m \times n}\), and \(b \in \reals^m\), then \[ AX+b \sim \mathcal{N}(Aμ_X+b,AΣ_XA^\TRANS). \] This remains true when \(AΣ_XA^\TRANS\) is singular.

  2. Marginals. Every subvector of a Gaussian random vector is Gaussian. Its mean and covariance are obtained by selecting the corresponding entries and blocks of the original mean and covariance.

  3. Sums. If \(X\) and \(Y\) are jointly Gaussian random vectors of the same dimension, then \(X+Y\) is Gaussian. In particular, independent Gaussian random vectors are jointly Gaussian, so their sum is Gaussian.

  4. Independence. If \(X\) and \(Y\) are jointly Gaussian, then \[ X \independent Y \quad\Longleftrightarrow\quad Σ_{XY}=0. \] Consequently, the components of a Gaussian random vector are mutually independent if and only if its covariance matrix is diagonal.

  5. Standardization. If \(X \sim \mathcal{N}(μ,Σ)\) and \(Σ\) is positive definite, then \[ Z=Σ^{-\frac12}(X-μ) \sim \mathcal{N}(0,I_n). \] This operation is also called whitening.

For any \(t \in \reals^m\), \[ t^\TRANS(AX+b)=(A^\TRANS t)^\TRANS X+t^\TRANS b, \] which is Gaussian by Definition 4.1. Its mean and variance are \[ t^\TRANS(Aμ_X+b) \quad\text{and}\quad t^\TRANS AΣ_XA^\TRANS t, \] respectively. The marginal and sum properties follow by applying appropriate selection or addition matrices to the relevant jointly Gaussian vector.

4.2.4 Independent components and Gaussian contours

Suppose \(Σ\) is positive definite and diagonal: \[ μ=\MATRIX{μ_1 \\ \vdots \\ μ_n}, \qquad Σ=\MATRIX{ σ_1^2 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & σ_n^2 }. \] Then \[ f_X(x) =\prod_{i=1}^n \frac{1}{\sqrt{2πσ_i^2}} \exp\left(-\frac{(x_i-μ_i)^2}{2σ_i^2}\right), \] which is the product of the marginal PDFs. Thus, the components are independent.

For a general positive-definite covariance matrix, the level sets of the PDF are \[ (x-μ)^\TRANS Σ^{-1}(x-μ)=k^2. \] These are ellipsoids centered at \(μ\). If \[ Σ=QΛQ^\TRANS, \] then the columns of \(Q\) give the principal directions of the ellipsoids, and their semi-axis lengths are \(k\sqrt{λ_i}\), where \(λ_i\) are the eigenvalues of \(Σ\).

For example, consider a two-dimensional Gaussian vector with \(σ_i^2=\VAR(X_i)>0\) and \[ ρ=\frac{\COV(X_1,X_2)}{σ_1σ_2}. \] Its covariance matrix is \[ Σ=\MATRIX{ σ_1^2 & ρσ_1σ_2 \\ ρσ_1σ_2 & σ_2^2 }. \] When \(\ABS{ρ}<1\), \[ \det(Σ)=(1-ρ^2)σ_1^2σ_2^2 \] and \[ Σ^{-1} =\frac{1}{(1-ρ^2)σ_1^2σ_2^2} \MATRIX{ σ_2^2 & -ρσ_1σ_2 \\ -ρσ_1σ_2 & σ_1^2 }. \] Therefore, \[ (x-μ)^\TRANS Σ^{-1}(x-μ) =\frac{1}{1-ρ^2} \left[ \frac{(x_1-μ_1)^2}{σ_1^2} -2ρ\frac{(x_1-μ_1)(x_2-μ_2)}{σ_1σ_2} +\frac{(x_2-μ_2)^2}{σ_2^2} \right]. \]

  • When \(ρ=0\), the ellipse axes are aligned with the coordinate axes.
  • When \(ρ>0\), the major axis has positive slope.
  • When \(ρ<0\), the major axis has negative slope.

When \(\ABS{ρ}=1\), the distribution is singular and is supported on the line \[ X_2-μ_2=ρ\frac{σ_2}{σ_1}(X_1-μ_1) \quad\text{almost surely}. \]

The following interactive figure illustrates these properties for a zero-mean bivariate Gaussian distribution. The standard deviations \(σ_1\) and \(σ_2\) control the spread along the two coordinates, while \(ρ\) controls the orientation and elongation of the contours. Recall that \(ρ=0\) corresponds to independent components in the jointly Gaussian case. The three curves are the contours \[ x^\TRANS Σ^{-1}x=k^2, \qquad k=1,2,3. \]

Figure 4.1: Interactive PDF and equal-Mahalanobis-distance contours for a zero-mean bivariate Gaussian distribution.

4.2.5 A linear Gaussian input-output model

Let \[ X \sim \mathcal{N}(μ_X,Σ_X) \] and suppose that the output is \[ Y=AX+b+W, \] where \(W \sim \mathcal{N}(0,Σ_W)\) is independent of \(X\). The affine-transformation and independence properties give \[ Y \sim \mathcal{N}(Aμ_X+b,AΣ_XA^\TRANS+Σ_W). \] Moreover, \(X\) and \(Y\) are jointly Gaussian, with \[ \MATRIX{X \\ Y} \sim \mathcal{N}\left( \MATRIX{μ_X \\ Aμ_X+b}, \MATRIX{ Σ_X & Σ_XA^\TRANS \\ AΣ_X & AΣ_XA^\TRANS+Σ_W } \right). \] In particular, \[ Σ_{XY}=Σ_XA^\TRANS. \] This calculation is the basic covariance propagation step for linear Gaussian models. A later chapter derives the conditional distribution of \(X\) given an observed value of \(Y\); see MMSE estimation.

4.2.6 Mixture of Gaussian distributions: BPSK over an AWGN channel

The preceding model assumes that the input is Gaussian. In a digital communication system, however, the transmitted symbol is discrete. Consider binary phase-shift keying (BPSK). Let \[ \PR(B=+1)=\PR(B=-1)=\frac12, \] and transmit the symbol \[ X=aB, \] where \(a>0\) is the signal amplitude. The received signal over an additive white Gaussian noise (AWGN) channel is \[ Y=X+W=aB+W, \] where \(W \sim \mathcal{N}(0,σ^2)\) is independent of \(B\).

Conditioned on the transmitted bit, the received signal is Gaussian: \[ Y \mid (B=+1) \sim \mathcal{N}(a,σ^2), \qquad Y \mid (B=-1) \sim \mathcal{N}(-a,σ^2). \] Define the zero-mean Gaussian density \[ φ_σ(u)=\frac{1}{\sqrt{2πσ^2}} \exp\left(-\frac{u^2}{2σ^2}\right). \] Marginalizing over the unobserved bit gives \[ f_Y(y) =\frac12 φ_σ(y-a)+\frac12 φ_σ(y+a). \] Thus, the channel output is a mixture of two Gaussian distributions rather than a Gaussian random variable. Its mean and variance are \[ \EXP[Y]=0, \qquad \VAR(Y)=a^2+σ^2. \] Nevertheless, these two moments do not make \(Y\) Gaussian. For \(a \neq 0\), \[ \EXP[Y^4] =a^4+6a^2σ^2+3σ^4 \neq 3(a^2+σ^2)^2, \] where the expression on the right is the fourth moment of a zero-mean Gaussian random variable with variance \(a^2+σ^2\).

The two conditional densities also determine the receiver. For equally likely bits, the log-likelihood ratio is \[ \log\frac{f_{Y\mid B}(y\mid +1)}{f_{Y\mid B}(y\mid -1)} =\frac{2ay}{σ^2}. \] Therefore, the maximum a posteriori decision rule is the threshold detector \[ \widehat B= \begin{cases} +1, & Y \ge 0, \\ -1, & Y<0. \end{cases} \]

Only the normalized separation \(a/σ\) determines the shape of the standardized mixture. In the following figure, \(σ=1\) and the slider changes \(a/σ\). At small separation the mixture appears nearly Gaussian and unimodal; at large separation its two components become visible.

Figure 4.2: Conditional Gaussian densities and the resulting BPSK output mixture as the normalized signal separation changes.