7 Moment Generating Functions
7.1 Moment Generating Functions
The moment generating function (MGF) of a random variable \(X\) is defined as \[ M_X(s) = \EXP[e^{sX}] \] for any \(s\in\mathbb C\) for which the expectation exists. Its region of convergence is \[ \mathcal R_X =\{s\in\mathbb C:\EXP[e^{\operatorname{Re}(s)X}]<∞\}. \]
When \(X\) is discrete, we have \[ M_X(s) = \sum_{x \in \ALPHABET X} e^{sx} p_X(x). \]
When \(X\) is continuous, we have \[ M_X(s) = \int_{-∞}^∞ e^{sx} f_X(x) \, dx. \]
Although most texts (including the textbook) restrict \(s\) to be real, we will interpret \(s\) as a complex number in the region of convergence \(\mathcal R_X\). Then:
- \(M_X(-s)\) is the bilateral Laplace transform of the probability distribution.
- \(M_X(-jω)\) is its Fourier transform, also called the characteristic function of \(X\).
Thus, specifying \(M_X\) on the imaginary axis uniquely specifies the probability distribution. When a PDF exists, it can be recovered by Fourier inversion under the usual inversion conditions.
The imaginary axis always belongs to \(\mathcal R_X\), because \(\ABS{e^{jωX}}=1\). For some distributions, the region of convergence contains no nonzero real numbers. For example, the region of convergence for a Cauchy random variable is the imaginary axis. We use \(M_X\) for the unified transform and do not introduce separate notation for its real- and imaginary-axis restrictions.
Example 7.1 Suppose \(X\) is a random variable which takes values \(\{0, 1, 2\}\) with probabilities \(\{\frac 12, \frac 13, \frac 16\}\). Then, \[\begin{align*} M_X(s) &= \EXP[e^{sX}] \\ &= \frac 12 e^{s 0} + \frac 13 e^{s 1} + \frac 16 e^{s 2} \\ &= \frac 12 + \frac 13 e^{s} + \frac 16 e^{2s}. \end{align*}\]
Example 7.2 Find the MGF of a Poisson random variable with parameter \(λ\).
\[\begin{align*} M_X(s) &= \EXP[e^{sX}] = \sum_{k=0}^{∞}e^{ks} \frac{λ^k e^{-λ}}{k!} \\ &= e^{-λ} \sum_{k=0}^{∞} \frac{(λe^s)^k}{k!} \\ &= e^{-λ} e^{λ e^{s}}. \end{align*}\]
Example 7.3 Find the MGF of an exponential random variable with parameter \(λ\).
\[\begin{align*} M_X(s) &= \EXP[e^{sX}] \\ &= \int_{0}^∞ e^{sx} λ e^{-λx} \, dx \\ &= λ \int_{0}^∞ e^{(s-λ)x} \, dx \\ &= \frac{λ}{λ-s}. \end{align*}\] The integral converges for \(\operatorname{Re}(s)<λ\).
Note that we could have looked up this result from the Laplace transform tables which show that \[ e^{at} \xleftrightarrow{\hskip 0.5em \mathcal{L}\hskip 0.5em } \frac{1}{s-a} \]
The MGF of common random variables is shown in Table 7.1.
| Random variable | Parameter(s) | MGF | Region of convergence |
|---|---|---|---|
| Bernoulli | \(p\) | \(1 - p + p e^s\) | \(s\in\mathbb C\) |
| Binomial | \((n,p)\) | \((1-p + p e^s)^n\) | \(s\in\mathbb C\) |
| Geometric | \(p\) | \(\dfrac{p e^s}{1 - (1-p)e^s}\) | \(\operatorname{Re}(s)<-\log(1-p)\) |
| Poisson | \(λ\) | \(\exp(λ (e^s - 1))\) | \(s\in\mathbb C\) |
| Uniform | \((a,b)\) | \(\dfrac{e^{sb} - e^{sa}}{s(b-a)}\) | \(s\in\mathbb C\) |
| Exponential | \(λ\) | \(\dfrac{λ}{λ-s}\) | \(\operatorname{Re}(s)<λ\) |
| Gaussian | \((μ,σ^2)\) | \(\exp\bigl(μ s + \frac 12 σ^2 s^2 \bigr)\) | \(s\in\mathbb C\) |
If the region of convergence \(\mathcal R_X\) contains an open neighborhood of the origin, then we can use the MGF to “generate the moments” of \(X\) as follows:
\(M_X(0) = 1\)
\(\dfrac{d}{ds} M_X(s) \biggr|_{s=0} = \EXP[X]\).
\(\dfrac{d^2}{ds^2} M_X(s) \biggr|_{s=0} = \EXP[X^2]\).
and in general \(\dfrac{d^k}{ds^k} M_X(s) \biggr|_{s=0} = \EXP[X^k]\).
Hence, by Taylor series expansion of \(M_X(s)\) within the radius of convergence, we get \[ M_X(s) = \sum_{k=0}^∞ \frac{\EXP[X^k]}{k!} s^k. \] Thus, when \(\mathcal R_X\) contains an open neighborhood of the origin, the moments determine the MGF in that neighborhood, and the MGF uniquely determines the probability distribution. Under this condition, the distribution of a random variable is completely characterized by its moments.
Without this condition, the moments need not uniquely determine the distribution. For example, the lognormal distribution has moments of every order but is not uniquely determined by them.
The first property follows from the definition: \[ M_X(0) = \EXP[e^{0 X}] = \EXP[1] = 1. \]
For the general derivative, we have \[\begin{align*} \frac{d^k}{ds^k} M_X(s) &= \int_{-∞}^∞ \frac{d^k}{ds^k} e^{sx} f_X(x) \, dx \\ &= \int_{-∞}^∞ x^k e^{sx} f_X(x) \, dx. \end{align*}\]
Therefore \[ \frac{d^k}{ds^k} M_X(s) \biggr|_{s=0} = \int_{-∞}^∞ x^k f_X(x) \, dx. \]
Example 7.4 Use the MGF of a Bernoulli random variable to find all the moments of \(X\).
From Table 7.1, we see that \[ M_X(s) = 1 - p + p e^{s}. \] Therefore,
\(\dfrac{d}{ds} M_X(s) = p e^s\).
\(\dfrac{d^2}{ds^2} M_X(s) = p e^s\).
and in general \(\dfrac{d^k}{ds^k} M_X(s) = p e^s\).
Thus,
\(\EXP[X] = \dfrac{d}{ds} M_X(s) \biggr|_{s=0} = p\).
\(\EXP[X^2] = \dfrac{d^2}{ds^2} M_X(s) \biggr|_{s=0} = p\).
and in general \(\EXP[X^k] = \dfrac{d^k}{ds^k} M_X(s) \biggr|_{s=0} = p\).
7.1.1 Cumulant generating functions
Suppose the region of convergence \(\mathcal R_X\) contains an open neighborhood of the origin. Since \(M_X(0)=1\), the transform is nonzero in a sufficiently small neighborhood of the origin. The cumulant generating function (CGF) is defined there by \[ K_X(s)=\log M_X(s), \qquad K_X(0)=0. \] For real \(s\), this is the ordinary real logarithm because \(M_X(s)>0\). For complex \(s\), we use the local branch of the logarithm containing \(s=0\).
The \(k\)-th cumulant of \(X\) is \[ κ_k(X)=\frac{d^k}{ds^k}K_X(s)\biggr|_{s=0}. \] The first four cumulants are \[\begin{align*} κ_1(X)&=\EXP[X]=μ,\\ κ_2(X)&=\VAR(X),\\ κ_3(X)&=\EXP[(X-μ)^3],\\ κ_4(X)&=\EXP[(X-μ)^4]-3\VAR(X)^2. \end{align*}\] Thus, the first cumulant is the mean, the second is the variance, and the third is the third central moment. Beginning with order four, cumulants and central moments are different.
If \(Z=X_1+\cdots+X_n\) and the random variables are independent, then \[ K_Z(s) =\log\left(\prod_{i=1}^nM_{X_i}(s)\right) =\sum_{i=1}^nK_{X_i}(s). \] Consequently, cumulants add under independent sums: \[ κ_k(Z)=\sum_{i=1}^nκ_k(X_i). \]
If \(Y=aX+b\), then \[ K_Y(s)=bs+K_X(as). \] Therefore, \[ κ_1(Y)=aκ_1(X)+b, \qquad κ_k(Y)=a^kκ_k(X),\quad k\ge2. \]
If \(X\sim\mathcal N(μ,σ^2)\), then \[ K_X(s)=μs+\frac12σ^2s^2. \] Hence, a Gaussian random variable has mean \(μ\), variance \(σ^2\), and all cumulants of order three and higher equal to zero.
If \(X\sim\operatorname{Pois}(λ)\), then \[ K_X(s)=λ(e^s-1). \] Thus, \(κ_k(X)=λ\) for every \(k\ge1\).
The CGF also appears naturally in exponential tail bounds. For \(s>0\) in the region of convergence, \[ \PR(X\ge a) \le \exp\bigl(-(sa-K_X(s))\bigr). \] Optimizing over \(s\) gives the Chernoff bound developed in the next chapter.
7.1.2 Moment generating functions and sums of independent random variables
Theorem 7.1 Suppose \(X_1, X_2, \dots, X_n\) are independent random variables defined on the same probability space. Let \[ Z = X_1 + X_2 + \dots + X_n \] Then, \[ M_Z(s) = M_{X_1}(s) M_{X_2}(s) \cdots M_{X_n}(s). \]
Furthermore, if the variables are identically distributed, then \[ M_Z(s) = (M_{X_1}(s))^n. \]
The proof follows immediately from the factorization property of independent random variables. We prove the result for \(n=2\): \[\begin{align*} M_Z(s) &=\EXP[e^{s(X_1+X_2)}]\\ &=\EXP[e^{sX_1}e^{sX_2}]\\ &=\EXP[e^{sX_1}]\EXP[e^{sX_2}]\\ &=M_{X_1}(s)M_{X_2}(s). \end{align*}\]
Theorem 7.1 is a very useful result. The following are immediate implications:
Sum of i.i.d. Bernoulli random variables is a Binomial random variable
Let \(X_i \sim \text{Ber}(p)\). Then \(M_{X_i}(s) = (1 - p + pe^s)\).
Let \(Z = \sum_{i=1}^n X_i\). Then \(M_Z(s) = (1 - p + pe^s)^n\).
Sum of independent Binomial random variables with the same \(p\) is a Binomial random variable.
Let \(X_i \sim \text{Binom}(m_i,p)\). Then \(M_{X_i}(s) = (1 - p + p e^s)^{m_i}\).
Let \(Z = \sum_{i=1}^n X_i\). Then \(M_Z(s) = (1 - p + p e^s)^M\), where \(M = \sum_{i=1}^n m_i\).
Sum of independent Poisson random variables is Poisson.
Let \(X_i \sim \text{Pois}(λ_i)\). Then \(M_{X_i}(s) = e^{λ_i(e^s - 1)}\).
Let \(Z = \sum_{i=1}^n X_i\). Then \(M_Z(s) = e^{λ(e^s - 1)}\), where \(λ = \sum_{i=1}^n λ_i\).
Sum of independent Gaussian random variables is Gaussian.
Let \(X_i \sim \mathcal N(μ_i, σ_i^2)\). Then, \(M_{X_i}(s) = \exp( μ_i s + \frac 12 σ_i^2 s^2)\).
Let \(Z = \sum_{i=1}^n X_i\). Then \(M_Z(s) = \exp(μ s + \frac 12 σ^2 s^2)\), where \[ μ = \sum_{i=1}^n μ_i \quad\text{and}\quad σ^2 = \sum_{i=1}^n σ_i^2. \]
7.1.3 Sum of a random number of random variables
Let \(N\) be a nonnegative integer-valued random variable that is independent of the i.i.d. random variables \(X_1, X_2, \ldots\), and define \[ S_N = \sum_{i=1}^N X_i, \qquad S_0 = 0. \] Assume that the moments and transforms below exist.
Conditional on \(N = n\), Theorem 7.1 gives \[ \EXP[e^{s S_N} \mid N = n] = (M_X(s))^n. \] Therefore, conditioning on \(N\) gives \[ M_{S_N}(s) = \EXP\bigl[(M_X(s))^N\bigr]. \]
The same conditioning gives the mean. Given \(N = n\), \[ \EXP[S_N \mid N = n] = \EXP[X_1 + \cdots + X_n] = n \EXP[X]. \] Hence \[ \EXP[S_N \mid N] = N \EXP[X], \] and taking expectation of both sides yields \[ \EXP[S_N] = \EXP[N] \EXP[X]. \]
If \(N \sim \operatorname{Pois}(λ)\) and \(X_i \sim \text{Ber}(p)\), then \(M_X(s) = 1 - p + p e^s\). Since \(\EXP[r^N] = \exp\bigl(λ(r - 1)\bigr)\), we obtain \[ M_{S_N}(s) = \exp\bigl(λ p (e^s - 1)\bigr). \] Thus \(S_N \sim \operatorname{Pois}(λ p)\). Independently keeping each of a Poisson number of items with probability \(p\) again yields a Poisson count.
Let \(N \sim \text{Geo}(p)\). Then \[ \EXP[r^N] = \frac{p r}{1 - (1 - p) r}, \] so \[ M_{S_N}(s) = \frac{p M_X(s)}{1 - (1 - p) M_X(s)}. \] If \(X_i \sim \text{Exponential}(λ)\), this simplifies to \[ M_{S_N}(s) = \frac{p λ}{p λ - s}, \] hence \(S_N \sim \text{Exponential}(p λ)\). If instead \(X_i \sim \text{Geo}(α)\), the same substitution yields \[ M_{S_N}(s) = \frac{p α e^s}{1 - (1 - p α) e^s}, \] hence \(S_N \sim \text{Geo}(p α)\). A geometric number of exponential waiting times is exponential; a geometric number of geometric counts is geometric.
7.2 The Central Limit Theorem
One of the ways in which MGFs are useful is that they allow us to understand the limiting behavior of sums of i.i.d. random variables.
A sequence of random variables \(\{X_n\}_{n \ge 1}\) is said to converge in distribution to a random variable \(X\) (denoted by \(X_n \xrightarrow{D} X\)) if \[ \lim_{n \to \infty} F_{X_n}(x) = F_X(x), \] for all \(x\) where \(F_X\) is continuous.
Example 7.5 Consider a sequence \(\{X_n\}_{n \ge 1}\) of random variables where \(X_n \sim \mathcal{N}(0,1/n)\). Show that \(X_n \xrightarrow{D} 0\).
Let \(F\) denote the CDF of the constant random variable \(X=0\), i.e., \[ F(x) = \begin{cases} 0, & x < 0 \\ 1, & x \ge 0 \end{cases} \]
Let \(Z\) denote the standard Gaussian random variable. Then \[ F_{X_n}(x) = \PR(X_n \le x) = \PR(Z \le \sqrt{n} x) \to \begin{cases} 1, & x > 0 \\ 0, & x < 0 \end{cases} \] Thus, \(F_{X_n}(x)\) converges to \(F(x)\) for all \(x \neq 0\). Recall that the definition of convergence in distribution does not require convergence of \(F_{X_n}(x)\) at points of discontinuity of \(F\). So, \(X_n \xrightarrow{D} 0\).
An important implication of convergence in distribution is that for any continuous bounded function \(g\) \[ \EXP[g(X_n)] \to \EXP[g(X)]. \] For this reason, convergence in distribution is sometimes called weak convergence.
The relationship between probability distributions and their transforms gives the following continuity theorem:
Consider a sequence of random variables \(\{X_n\}_{n \ge 1}\). For ease of notation, let \((F_n,M_n)\) denote the CDF and transform of \(X_n\).
If \(F_n\to F\) for some CDF \(F\) with transform \(M\), then \[ M_n(jω)\to M(jω), \qquad ω\in\reals. \]
Conversely, suppose \[ \psi(ω)=\lim_{n\to∞}M_n(jω) \] exists for every \(ω\in\reals\) and is continuous at \(ω=0\). Then \(\psi(ω)=M(jω)\) for some probability distribution with CDF \(F\), and \(F_n\to F\).
The proof is a bit involved and is omitted. The convergence statement uses the imaginary axis, where \(\ABS{e^{jωX}}=1\). Convergence of the transform away from the imaginary axis generally requires additional conditions.
7.2.1 Two limit theorems
We now prove two limit theorems using the transform.1
1 We first perform the transform calculations for a general \(s\) in the region of convergence. We restrict to \(s=jω\) only when applying the continuity theorem.
Theorem 7.2 (A law of large numbers) Let \(\{X_n\}_{n \ge 1}\) be a sequence of i.i.d. random variables with finite mean \(μ\). Then, their partial sums \(S_n = X_1 + \cdots + X_n\) satisfy \[ \frac{1}{n} S_n \xrightarrow{D} μ \]
The theorem asserts that \[ \lim_{n\to∞}\PR(n^{-1}S_n \le x) = \begin{cases} 0 & \hbox{if } x < μ, \\ 1 & \hbox{if } x > μ. \end{cases} \] No condition is required at \(x=μ\), where the limiting CDF is discontinuous.
Let \(M_X\) denote the common transform of the \(X_i\). From Theorem 7.1, we know that \[ M_{n^{-1} S_n}(s) = M_X(s/n)^n, \qquad s\in\mathcal R_X. \]
For any fixed \(s\in\mathcal R_X\), the expansion as \(s/n\to0\) within the region of convergence gives \[ M_X(s/n) = 1 + \frac{μs}{n} + o\left(\frac{1}{n}\right). \] Therefore, \[M_{n^{-1} S_n}(s) = \left(1 + \frac{μs}{n} + o\left(\frac{1}{n}\right) \right)^n \to \exp(μs) \quad \text{as} \quad n \to ∞, \] which is the transform of the constant random variable \(μ\). In particular, taking \(s=jω\) gives \[ M_{n^{-1}S_n}(jω)\to e^{jμω}. \] The continuity theorem therefore gives \(n^{-1}S_n\xrightarrow{D}μ\).
We will show later that convergence in distribution to a constant implies convergence in probability. Therefore, the above implies the weak law of large numbers.
The above result shows that when \(n\) is large, the sum \(S_n\) is approximately \(nμ\). The Central Limit Theorem describes the size and distribution of the fluctuations around \(nμ\). When \(X_n\) has finite variance \(σ^2\):
- \(S_n - n μ\) is about as big as \(\sqrt{n}\).
- Irrespective of the distribution of \(X_n\), \((S_n - n μ)/\sqrt{n}\) converges in distribution to a normal distribution with variance \(σ^2\).
We prove the result below.
Theorem 7.3 (Central Limit Theorem) Let \(\{X_n\}_{n \ge 1}\) be i.i.d. random variables with mean \(μ\) and finite non-zero variance \(σ^2\). Then, their partial sums \(S_n = X_1 + \cdots + X_n\) satisfy \[ \frac{S_n - n μ}{\sqrt{n}\, σ} \xrightarrow{D} {\cal N}(0,1). \]
The proof idea is similar to that of Theorem 7.2. Define the normalized random variables \(Y_i=(X_i-μ)/σ\), which have mean zero and unit variance. Let \(M_Y\) denote their common transform. Its expansion within the region of convergence is \[ M_Y(s)=1+\frac12s^2+o(\ABS{s}^2), \qquad s\to0. \] Moreover, observe that \[ Z_n \coloneqq \frac{S_n - n μ }{\sqrt{n}\, σ} = \frac{1}{\sqrt{n}} \sum_{i=1}^n Y_i. \] Therefore, for any fixed \(s\in\mathcal R_Y\), Theorem 7.1 gives \[ M_{Z_n}(s) = M_Y(s/\sqrt{n})^n = \left(1+\frac{s^2}{2n}+o\left(\frac{1}{n}\right)\right)^n \to \exp(\tfrac12s^2). \] Taking \(s=jω\) gives \[ M_{Z_n}(jω)\to\exp(-\tfrac12ω^2), \] which is the transform of \(\mathcal N(0,1)\) on the imaginary axis. The continuity theorem gives \(Z_n\xrightarrow{D}\mathcal N(0,1)\).
The central limit theorem is one of the cornerstones of probability theory. The earliest statement of the result goes back to de Moivre (1773), but there was little follow-up until Laplace’s Théorie analytique des probabilités (1812). The term “central limit theorem” is due to Pólya (1920), who chose the name because the theorem plays a central role in probability theory.
7.3 Moment generating functions of random vectors
Let \(X\in\reals^n\) be a random vector. For \(s\in\mathbb C^n\), define \[ M_X(s)=\EXP[e^{s^\TRANS X}] \] whenever the expectation exists. Its region of convergence is \[ \mathcal R_X =\left\{ s\in\mathbb C^n: \EXP[e^{\operatorname{Re}(s)^\TRANS X}]<∞ \right\}. \] In particular, \(j\reals^n\subseteq\mathcal R_X\).
If \(\mathcal R_X\) contains an open neighborhood of the origin, partial derivatives of the transform generate the joint moments. In particular, \[ \nabla M_X(0)=\EXP[X]=μ_X \] and \[ \nabla^2M_X(0)=\EXP[XX^\TRANS]. \] Therefore, the covariance matrix is \[ \boxed{ Σ_X=\nabla^2M_X(0)-\nabla M_X(0)\nabla M_X(0)^\TRANS. } \]
When \(\mathcal R_X\) contains an open neighborhood of the origin, the cumulant generating function of a random vector is \[ K_X(s)=\log M_X(s). \] Its first two derivatives have the particularly simple interpretations \[ \nabla K_X(0)=μ_X, \qquad \nabla^2K_X(0)=Σ_X. \]
If \(Y=AX+b\), where \(A\in\reals^{m\times n}\) and \(b\in\reals^m\), then for \(t\in\mathbb C^m\) for which the expectations exist, \[ M_Y(t) =e^{t^\TRANS b}M_X(A^\TRANS t). \]
If the components \(X_1,\ldots,X_n\) are independent, then \[ M_X(s) =\prod_{i=1}^n M_{X_i}(s_i), \qquad s\in\mathcal R_X. \] Conversely, factorization on the imaginary subspace \(j\reals^n\) implies that the components are independent, because the transform on \(j\reals^n\) uniquely determines the joint distribution.
If \(X\sim\mathcal N(μ,Σ)\) is a Gaussian random vector, then \[ \boxed{ M_X(s) =\exp\left(s^\TRANS μ+\frac12s^\TRANS Σs\right), \qquad s\in\mathbb C^n. } \] Thus, the transform of a Gaussian random vector is completely determined by its mean vector and covariance matrix, just as its probability distribution is.