ECSE 506: Stochastic Control and Decision Theory
Aditya Mahajan
Winter 2022
About  Lectures  Notes  Coursework
1 Change of measure of a single random variable.
 Theorem 1

Let \((\Omega, \mathcal F, P)\) be a probability space and \(\Lambda\) be an almost surely nonnegative random variable such that \(\EXP[\Lambda] = 1\). For any \(A \in \mathcal F\), define \[ P^\dagger(A) = \int_A \Lambda(\omega) dP(\omega). \] Then,
 \(P^\dagger\) is a probability measure.
 For any random variable \(X\), \[ \EXP^\dagger[X] = \EXP[ \Lambda X]. \]
 If \(\Lambda\) is almost surely positive, then \[ \EXP[X] = \EXP^\dagger \left[ \frac{X}{\Lambda} \right]. \]
Proof
By definition. \(P^\dagger(\emptyset) = 0\) and \(P^\dagger(\Omega) = \EXP[ \Lambda] = 1\). Since \(\Lambda\) is almost surely nonnegative, \(P^\dagger(A) \ge 0\). Hence, \(P^\dagger\) is a probability measure.
The second and the third part follow from observing that \[ dP^\dagger(\omega) = \Lambda(\omega) dP(\omega). \]
Given two measures \(\mu\) and \(\nu\) on a measurable space \((\Omega, \mathcal F)\), we say that the measure \(\mu\) is absolutely continuous with respect to \(\nu\) (denoted by \(\mu \ll \nu\)) if for any \(A \in \mathcal F\), \[ \nu(A) = 0 \implies \mu(A) = 0. \]
 Theorem 2

(RadonNikodym). Given two probability measures \(P\) and \(P^\dagger\) on a measurable space, if \(P^\dagger\) is absolutely continuous with respect to \(P\), then there exists an almost surely positive random variable \(\Lambda\) such that \(\EXP[\Lambda] = 1\) and for any \(A \in \mathcal F\), \[ P^\dagger(A) = \int_A \Lambda(\omega) dP(\omega). \] Such a \(\Lambda\) is called the RadonNikodym derivative of \(P^\dagger\) with respect to \(P\), and is written as \[ \Lambda = \frac{ dP^\dagger } {dP}. \]
 Remark

The RadonNikodym theorem provides the reverse property of Theorem 1. Given two measures \(μ \ll ν\), \[ \int_{A} f dν = \int_A f \frac{dν}{dμ} dμ. \] Thus, in Theorem 1, we are constructing a new probaility measure \(P^\dagger\) such that \(dP^\dagger/dP = Λ\).
The RadonNikodym Theorem is typically stated for \(σ\)finite measures. The above statement is a specialization of RadonNikodym Theorem to probability measures.
In statistical signal processing literature, the RadonNikodym derivative is sometimes known as the likelihood ratio. In the reinforcement learning literature, it is called importance sampling.
The density of a random variable is the RadonNikodym derivative with respect to the Lebesgue measure.
The RadonNikodym derivative satisfies the product rule. If \(μ \ll ν \ll λ\), then \[ \frac {dμ}{dλ} = \frac {dμ}{dν} \frac {dν}{dλ}, \quad λ~\text{a.s.}. \]
The KullbackLeibler divergence between two probability measures \(P\) and \(Q\) defined on \((\Omega, \mathcal F)\) may be written as \[ D_{\text{KL}}( P \ Q) = \int_\Omega \log \left ( \frac {dP}{dQ} \right) dP. \]
2 Conditional expectation under change of measure
 Theorem 3

Consider two probability measures \(P\) and \(P^\dagger\) on \((Ω, \mathcal F)\) such that \(P^\dagger \ll P\). Let \(Λ\) denote the RadonNikodym derivative of \(P^\dagger\) with respect to \(P\) and \(\mathcal G\) be any sub sigmafield of \(\mathcal F\). Then, for any random variable \(X\) \[ \EXP^\dagger[ X  \mathcal G ] = \dfrac{ \EXP[ Λ X  \mathcal G ] } { \EXP [ Λ  \mathcal G ] }, \quad P^\dagger~\text{a.s.} \]
Proof
Let \(G \in \mathcal G\). Then:
\[\begin{align*} \int_G \EXP[ Λ X  \mathcal G] dP &\stackrel{(a)}= \int_G Λ X dP \\ &\stackrel{(b)}= \int_G X dP^\dagger \\ &\stackrel{(c)}= \int_G \EXP^\dagger[ X  \mathcal G] dP^\dagger \\ &\stackrel{(d)}= \int_G \EXP^\dagger[ X  \mathcal G] Λ dP \\ &\stackrel{(e)}= \int_G \EXP[ \EXP^\dagger[ X  \mathcal G] Λ  \mathcal G] dP \\ &\stackrel{(f)}= \int_G \EXP^\dagger[ X  \mathcal G] \EXP[ Λ  \mathcal G] dP \\ \end{align*}\] where (a), (c), and (e) follow from the definition of conditional expectation, (b) and (d) follow from change of measures, and (f) follows because \(\EXP^\dagger[ X  \mathcal G]\) is \(\mathcal G\)measurable. Thus,
\[ \EXP[ Λ X  \mathcal G ] = \EXP^\dagger[ X  \mathcal G ] \EXP[ Λ  \mathcal G]. \]
3 Change of measure for a process
Consider a probability space \((Ω, \mathcal F)\) and let \(P\) and \(P^\dagger\) be two probability measures on \((Ω, \mathcal F)\) such that \(P^\dagger \ll P\). Let \(Λ\) denote the RadonNikodym derivative of \(P^\dagger\) with respect to \(P\).
Let \(\{\mathcal F_t\}_{t \ge 0}\) be a filtration on \((Ω, \mathcal F)\). Then, we can define the RadonNikodym derivative process \[ Λ_t = \EXP[ Λ  \mathcal F_t ]. \]
 Theorem 4

The RadonNikodym derivative process \(\{Λ_t\}_{t \ge 0}\) is a martingale with respect to \(\{\mathcal F_t\}_{t \ge 0}\), i.e., for any \(s \le t\), \[ \EXP[ Λ_t  \mathcal F_s ] = Λ_s. \]
Let \(X_t\) be an \(\mathcal F_t\) measurable random variable. Then \[ \EXP^\dagger[X_t] = \EXP[Λ X_t ] = \EXP[ Λ_t X_t ]. \]
Thus, \(Λ_t\) may be viewed as \(\dfrac {dP^\dagger}{dP} \Bigg_{\mathcal F_t}\).
Let \(X_t\) be an \(\mathcal F_t\) measurable random varaible. Then for any \(s < t\), \[ \EXP^\dagger[X_t  \mathcal F_s ] = \dfrac{1}{Λ_s} \EXP[ Λ_t X_t  \mathcal F_s ] . \]
An immediate implication of Theorem 4 is the following.
 Corollary

A process \(\{X_t\}_{t \ge 0}\) is a \(P^\dagger\)martingale with respect to \(\{\mathcal F_t\}_{t \ge 0}\) if and only if the process \(\{ Λ_t X_t \}_{t \ge 0}\) is a \(P\)martingale.
Proof
The fact RadonNikodym derivate process is a martingale immediately follows from the towering property of conidtional expectation:
\[ \EXP[ Λ_t  \mathcal F_s ] = \EXP[ \EXP[ Λ  \mathcal F_t ]  \mathcal F_s ] = \EXP[ Λ  \mathcal F_s ] = Λ_s. \]
By definition of RadonNikodym derivative, \(\EXP^\dagger[X_t] = \EXP[Λ X_t]\). Now, by the towering property of conditional expectation, we have \[ \EXP[Λ X_t ] = \EXP[ \EXP[ Λ X_t  \mathcal F_t ] ] = \EXP[ X_t \EXP[ Λ  \mathcal F_t ] ] = \EXP [Λ_t X_t]. \] This proves the second part.
To prove the third part, Theorem 3 implies that
\[\begin{equation} \EXP^\dagger[ X_t  \mathcal F_s ] = \frac{ \EXP[ Λ X_t  \mathcal F_s ]} { \EXP[ Λ  \mathcal F_s ] } = \frac{ \EXP[ Λ X_t  \mathcal F_s ]} { Λ_s }. \label{eq:step1} \end{equation}\]
Now, consider the numerator:
\[ \EXP[ Λ X_t  F_s ] = \EXP[ \EXP [ Λ X_t  \mathcal F_t ]  \mathcal F_s ] = \EXP [ X_t \EXP[ Λ  \mathcal F_t ] ] = \EXP [ X_t Λ_t ] . \] Substituting this in \eqref{eq:step1} completes the proof of the third part.
This entry was last updated on 13 Jun 2020