# Course Notes

# About the course

These notes started as the lecture notes for ECSE 506 (Stochastic Control and Decision Theory) that I teach in the Winter term of every even year. These notes are not meant to be exhaustive; rather my focus is to convey the key ideas in their simplest form. For a more exhaustive treatment of the subject, please refer to the reference books mentioned below.

#### Qualitative properties

#### Approximations

#### Linear Systems

If you find any typos/mistakes in the notes, please let me know. Pull requests are welcome.

## Reference books

Kumar and Varaiya,

*Stochastic Systems: Estimation, Identification, and Adaptive Control,*Prentice Hall, 1986. Reprinted by SIAM 2015

A gentle introduction which emphaisizes the key conceptual ideas.Bertsekas,

*Dynamic programming and optimal control*, vol 1 and 2, Athena Publications, 2005.

Perhaps the most comprehensive book of different topics in dynamic programming.Puterman,

*Markov decision processes: discrete time dynamic programming*, Wiley 1994.

Excellent source algorithms for perfectly observed systems, in particular, infinite horizon dynamic programs.Ross,

*Introduction to Stochastic Dynamic Programming,*Academic Press, 1983.

Excellent introduction to dynamic programming, from the point-of-view of applied mathematics.Dernardo,

*Dynamic Programming: Models and Applications,*Prentice Hall, 1982.

Excellent introduction to dynamic programming, from the point-of-view of operations research.Powell,

*Approximate Dynamic Programming*, John Wiley and Sons, 2011.

Comprehensive overview of approximate dynamic programmingKrishnamurty,

*Partially Observable Markov Decision Processes*, Cambridge University Press, 2016.

Comprehensive overview of POMDPsSargent and Stachurski,

*Dynamic Programming*, 2023.

Nice summary of DP ideas applied to economic models. Good mix of theory and numerical examples.Kochenderfer, Wheeler, and Wray,

*Algorithms for decision making*, MIT Press, 2022.

Broad introduction to decision making under uncertainty. Lots of nice examples.

## How to cite these notes

To cite these lecture notes, please use:

```
@misc{506notes,
author = {Aditya Mahajan},
title = {Lecture notes on Stochastic Control and Decision Theory},
year = {2023},
howpublished = "\url{https://adityam.github.io/stochastic-control/}",
}
```