ECSE 506: Stochastic Control and Decision Theory

Aditya Mahajan
Winter 2020

About  |  Lectures  |  Notes  |  Coursework


ENGTR 2100 (Tuesday and Thursday 10:05am to 11:25am)
Starting from 31 March 2020, the class will be held online on Zoom. Use the following link to join: You can also join via phone: +1 (438) 809-7799 (meeting id: 304330076)

Office hours

Thursday, 2:00pm to 3:00pm, MC 533.

Course Outline

Modeling of stochastic control systems, controlled Markov processes, dynamic programming, imperfect and delayed observations, linear quadratic and Gaussian (LQG) systems, team theory, information structures, static and dynamic teams, dynamic programming for teams, multi-armed bandits.

Course pre-requisites
  • A graduate course on Probability (ECSE 509 or equivalent) is a required pre-requisite.
  • A graduate course on Stochastic processes (ECSE 510 or equivalent) is a recommended co-requisite.
Grading policy
  • 20% 35% weekly assignments. Depending on the availability of graders, only a few questions, at random, will be graded. Late submissions will be penalized by 10% per day.
  • 25% 35% mid-terms. In class, March 10.
  • 20% 30% term project. A month long project to be done in groups of two individually. Present one or two papers on any topic of your interest related to the material covered in class. A 10 to 15 page report. and a 10 minute in-class presentations.
  • 35% final. During the exam period, scheduled by the university.

For most of the course, I will loosely follow the notation of Kumar and Varaiya. Many of the examples done in class are taken from Bertsekas. Some material on numerical implementation is from Puterman.

Reference books
  • Ross, Introduction to Stochastic Dynamic Programming, Academic Press, 1983.

    Excellent introduction to dynamic programming, from the point-of-view of applied mathematics.

  • Dernardo, Dynamic Programming: Models and Applications, Prentice Hall, 1982.

    Excellent introduction to dynamic programming, from the point-of-view of operations research.

  • Powell, Approximate Dynamic Programming, John Wiley and Sons, 2011.

    Comprehensive overview of approximate dynamic programming