Stochastic Control
Assignments
Assignment 8
Course Outline
Stochastic Optimization
1
Introduction
2
Dealing with observations
3
Certainty equivalence
4
Interchange arguments
MDPs
5
Finite horizon MDPs
6
Optimal gambling
7
Inventory Management
8
Some simplifying structures
9
Monotonicity of value function and optimal policies
10
Power-delay tradeoff
11
Continuous state spaces
12
Service migration in mobile edge computing
13
Reward Shaping
14
Optimal stopping
15
Infinite horizon MDPs
16
MDP algorithms
17
Inventory management (revisted)
18
Computational complexity of value iteration
19
Thirfty and equalizing policies
20
Linear programming formulation
21
Lipschitz MDPs
22
Periodic MDPs
23
Policy Gradient Theorem
POMDPs
24
Introduction
25
Sequential hypothesis testing
26
When to observe a Markov chain
27
Sequential hypothesis testing for Markov chains
Approx DP
28
Approximate dynamic programming
29
Upper bounds on policy loss
30
Model approximation
Risk sensitive MDPs
31
Risk Sensitive Utility
32
Risk Sensitive MDPs
33
DP for dynamic risk measures
Linear systems
34
Linear quadratic regulation
35
Large scale systems
36
Linear filtering
Stochastic Approximation
37
Stochastic approximation
38
Rate of convergence for stochastic approximation
RL
39
The learning setup
40
Q-Learning
Dec-POMDPs
41
Designer’s Approach
Analysis Appendix
42
Convergence of sequences
43
Inequalities
Probability Appendix
44
Convergence of random variables
45
Sub-Gaussian random variables
46
Change of Measure
47
Integral Probablity Metrics
48
Markov chains
49
Martingales
50
Stochastic stability
Linear Algebra Appendix
51
Some useful matrix relationships
52
Positive definite matrices
53
Matrix trace
54
Infinite product of matrices
55
Singular value decomposition
56
Vector, Banach, and Hilbert spaces
57
Reproducing Kernel Hilbert Space
Convexity Appendix
58
Convex sets and convex functions
59
Convex Duality
References
Assignments
Assignment 1
Assignment 2
Assignment 3
Assignment 4
Assignment 5
Assignment 6
Assignment 7
Assignment 8
60
Grading rubric for the report
Assignments
Assignment 8
Assignment 8
Author
Affiliation
Aditya Mahajan
McGill University
Updated
July 15, 2026
Exercise
16.2
from the notes on
MDP algorithms
.
Assignment 7
60
Grading rubric for the report