ECSE 506: Stochastic Control and Decision Theory

Aditya Mahajan
Winter 2022

About  |  Lectures  |  Notes  |  Coursework

Prelim: Change of Measure

1 Change of measure of a single random variable.

Theorem 1

Let \((\Omega, \mathcal F, P)\) be a probability space and \(\Lambda\) be an almost surely non-negative random variable such that \(\EXP[\Lambda] = 1\). For any \(A \in \mathcal F\), define \[ P^\dagger(A) = \int_A \Lambda(\omega) dP(\omega). \] Then,

  • \(P^\dagger\) is a probability measure.
  • For any random variable \(X\), \[ \EXP^\dagger[X] = \EXP[ \Lambda X]. \]
  • If \(\Lambda\) is almost surely positive, then \[ \EXP[X] = \EXP^\dagger \left[ \frac{X}{\Lambda} \right]. \]

Proof

By definition. \(P^\dagger(\emptyset) = 0\) and \(P^\dagger(\Omega) = \EXP[ \Lambda] = 1\). Since \(\Lambda\) is almost surely non-negative, \(P^\dagger(A) \ge 0\). Hence, \(P^\dagger\) is a probability measure.

The second and the third part follow from observing that \[ dP^\dagger(\omega) = \Lambda(\omega) dP(\omega). \]


Given two measures \(\mu\) and \(\nu\) on a measurable space \((\Omega, \mathcal F)\), we say that the measure \(\mu\) is absolutely continuous with respect to \(\nu\) (denoted by \(\mu \ll \nu\)) if for any \(A \in \mathcal F\), \[ \nu(A) = 0 \implies \mu(A) = 0. \]

Theorem 2

(Radon-Nikodym). Given two probability measures \(P\) and \(P^\dagger\) on a measurable space, if \(P^\dagger\) is absolutely continuous with respect to \(P\), then there exists an almost surely positive random variable \(\Lambda\) such that \(\EXP[\Lambda] = 1\) and for any \(A \in \mathcal F\), \[ P^\dagger(A) = \int_A \Lambda(\omega) dP(\omega). \] Such a \(\Lambda\) is called the Radon-Nikodym derivative of \(P^\dagger\) with respect to \(P\), and is written as \[ \Lambda = \frac{ dP^\dagger } {dP}. \]

Remark
  • The Radon-Nikodym theorem provides the reverse property of Theorem 1. Given two measures \(μ \ll ν\), \[ \int_{A} f dν = \int_A f \frac{dν}{dμ} dμ. \] Thus, in Theorem 1, we are constructing a new probaility measure \(P^\dagger\) such that \(dP^\dagger/dP = Λ\).

  • The Radon-Nikodym Theorem is typically stated for \(σ\)-finite measures. The above statement is a specialization of Radon-Nikodym Theorem to probability measures.

  • In statistical signal processing literature, the Radon-Nikodym derivative is sometimes known as the likelihood ratio. In the reinforcement learning literature, it is called importance sampling.

  • The density of a random variable is the Radon-Nikodym derivative with respect to the Lebesgue measure.

  • The Radon-Nikodym derivative satisfies the product rule. If \(μ \ll ν \ll λ\), then \[ \frac {dμ}{dλ} = \frac {dμ}{dν} \frac {dν}{dλ}, \quad λ~\text{a.s.}. \]

  • The Kullback-Leibler divergence between two probability measures \(P\) and \(Q\) defined on \((\Omega, \mathcal F)\) may be written as \[ D_{\text{KL}}( P \| Q) = \int_\Omega \log \left ( \frac {dP}{dQ} \right) dP. \]

2 Conditional expectation under change of measure

Theorem 3

Consider two probability measures \(P\) and \(P^\dagger\) on \((Ω, \mathcal F)\) such that \(P^\dagger \ll P\). Let \(Λ\) denote the Radon-Nikodym derivative of \(P^\dagger\) with respect to \(P\) and \(\mathcal G\) be any sub sigma-field of \(\mathcal F\). Then, for any random variable \(X\) \[ \EXP^\dagger[ X | \mathcal G ] = \dfrac{ \EXP[ Λ X | \mathcal G ] } { \EXP [ Λ | \mathcal G ] }, \quad P^\dagger~\text{a.s.} \]

Proof

Let \(G \in \mathcal G\). Then:

\[\begin{align*} \int_G \EXP[ Λ X | \mathcal G] dP &\stackrel{(a)}= \int_G Λ X dP \\ &\stackrel{(b)}= \int_G X dP^\dagger \\ &\stackrel{(c)}= \int_G \EXP^\dagger[ X | \mathcal G] dP^\dagger \\ &\stackrel{(d)}= \int_G \EXP^\dagger[ X | \mathcal G] Λ dP \\ &\stackrel{(e)}= \int_G \EXP[ \EXP^\dagger[ X | \mathcal G] Λ | \mathcal G] dP \\ &\stackrel{(f)}= \int_G \EXP^\dagger[ X | \mathcal G] \EXP[ Λ | \mathcal G] dP \\ \end{align*}\] where (a), (c), and (e) follow from the definition of conditional expectation, (b) and (d) follow from change of measures, and (f) follows because \(\EXP^\dagger[ X | \mathcal G]\) is \(\mathcal G\)-measurable. Thus,

\[ \EXP[ Λ X | \mathcal G ] = \EXP^\dagger[ X | \mathcal G ] \EXP[ Λ | \mathcal G]. \]

3 Change of measure for a process

Consider a probability space \((Ω, \mathcal F)\) and let \(P\) and \(P^\dagger\) be two probability measures on \((Ω, \mathcal F)\) such that \(P^\dagger \ll P\). Let \(Λ\) denote the Radon-Nikodym derivative of \(P^\dagger\) with respect to \(P\).

Let \(\{\mathcal F_t\}_{t \ge 0}\) be a filtration on \((Ω, \mathcal F)\). Then, we can define the Radon-Nikodym derivative process \[ Λ_t = \EXP[ Λ | \mathcal F_t ]. \]

Theorem 4
  • The Radon-Nikodym derivative process \(\{Λ_t\}_{t \ge 0}\) is a martingale with respect to \(\{\mathcal F_t\}_{t \ge 0}\), i.e., for any \(s \le t\), \[ \EXP[ Λ_t | \mathcal F_s ] = Λ_s. \]

  • Let \(X_t\) be an \(\mathcal F_t\) measurable random variable. Then \[ \EXP^\dagger[X_t] = \EXP[Λ X_t ] = \EXP[ Λ_t X_t ]. \]

    Thus, \(Λ_t\) may be viewed as \(\dfrac {dP^\dagger}{dP} \Bigg|_{\mathcal F_t}\).

  • Let \(X_t\) be an \(\mathcal F_t\) measurable random varaible. Then for any \(s < t\), \[ \EXP^\dagger[X_t | \mathcal F_s ] = \dfrac{1}{Λ_s} \EXP[ Λ_t X_t | \mathcal F_s ] . \]

An immediate implication of Theorem 4 is the following.

Corollary

A process \(\{X_t\}_{t \ge 0}\) is a \(P^\dagger\)-martingale with respect to \(\{\mathcal F_t\}_{t \ge 0}\) if and only if the process \(\{ Λ_t X_t \}_{t \ge 0}\) is a \(P\)-martingale.

Proof

The fact Radon-Nikodym derivate process is a martingale immediately follows from the towering property of conidtional expectation:

\[ \EXP[ Λ_t | \mathcal F_s ] = \EXP[ \EXP[ Λ | \mathcal F_t ] | \mathcal F_s ] = \EXP[ Λ | \mathcal F_s ] = Λ_s. \]

By definition of Radon-Nikodym derivative, \(\EXP^\dagger[X_t] = \EXP[Λ X_t]\). Now, by the towering property of conditional expectation, we have \[ \EXP[Λ X_t ] = \EXP[ \EXP[ Λ X_t | \mathcal F_t ] ] = \EXP[ X_t \EXP[ Λ | \mathcal F_t ] ] = \EXP [Λ_t X_t]. \] This proves the second part.

To prove the third part, Theorem 3 implies that

\[\begin{equation} \EXP^\dagger[ X_t | \mathcal F_s ] = \frac{ \EXP[ Λ X_t | \mathcal F_s ]} { \EXP[ Λ | \mathcal F_s ] } = \frac{ \EXP[ Λ X_t | \mathcal F_s ]} { Λ_s }. \label{eq:step-1} \end{equation}\]

Now, consider the numerator:

\[ \EXP[ Λ X_t | F_s ] = \EXP[ \EXP [ Λ X_t | \mathcal F_t ] | \mathcal F_s ] = \EXP [ X_t \EXP[ Λ | \mathcal F_t ] ] = \EXP [ X_t Λ_t ] . \] Substituting this in \eqref{eq:step-1} completes the proof of the third part.

This entry was last updated on 13 Jun 2020