Stochastic Control and Decision Theory

Course Notes for ECSE 506 (McGill University)

Author

Affiliation

Aditya Mahajan

McGill University

About the course

These notes started as the lecture notes for ECSE 506 (Stochastic Control and Decision Theory) that I teach in the Winter term of every even year. These notes are not meant to be exhaustive; rather my focus is to convey the key ideas in their simplest form. For a more exhaustive treatment of the subject, please refer to the reference books mentioned below.

If you find any typos/mistakes in the notes, please let me know. Pull requests are welcome.

Reference books

• Kumar and Varaiya, Stochastic Systems: Estimation, Identification, and Adaptive Control, Prentice Hall, 1986. Reprinted by SIAM 2015
  A gentle introduction which emphaisizes the key conceptual ideas.

• Bertsekas, Dynamic programming and optimal control, vol 1 and 2, Athena Publications, 2005.
  Perhaps the most comprehensive book of different topics in dynamic programming.

• Puterman, Markov decision processes: discrete time dynamic programming, Wiley 1994.
  Excellent source algorithms for perfectly observed systems, in particular, infinite horizon dynamic programs.

• Ross, Introduction to Stochastic Dynamic Programming, Academic Press, 1983.
  Excellent introduction to dynamic programming, from the point-of-view of applied mathematics.

• Dernardo, Dynamic Programming: Models and Applications, Prentice Hall, 1982.
  Excellent introduction to dynamic programming, from the point-of-view of operations research.

• Powell, Approximate Dynamic Programming, John Wiley and Sons, 2011.
  Comprehensive overview of approximate dynamic programming

• Krishnamurty, Partially Observable Markov Decision Processes, Cambridge University Press, 2016.
  Comprehensive overview of POMDPs

• Sargent and Stachurski, Dynamic Programming, 2023.
  Nice summary of DP ideas applied to economic models. Good mix of theory and numerical examples.

• Kochenderfer, Wheeler, and Wray, Algorithms for decision making, MIT Press, 2022.
  Broad introduction to decision making under uncertainty. Lots of nice examples.

How to cite these notes

To cite these lecture notes, please use:

@misc{506notes,
  author        = {Aditya Mahajan},
  title         = {Lecture notes on Stochastic Control and Decision Theory},
  year          = {2024},
  howpublished  = "\url{https://adityam.github.io/stochastic-control/}",
}