References

Dynamic programming (DP) is a fairly powerful and yet general framework that finds its use in many disciplines. Optimal control is not the only one. But in this overview of the literature we deliberately focus on the DP references with optimal control flavour.

Our introductory treatment was based almost exclusively on the (also just introductory) Chapter 6 in (Lewis, Vrabie, and Syrmo 2012). Electronic version of the book is freely available on the author’s webpage.

Comparable introduction is provided in (Kirk 2004). Although it does not appear to be legally available for free in an electronic form, its reprint by a low-cost publisher makes it an affordable (and recommendable) classic reference. Another classic (Anderson and Moore 2007) actually uses dynamic programming as the key technique to derive all those LQ-optimal regulation and tracking results. A few copies of this book are available in the faculty library at NTK. The authors also made an electronic version available for free on their website.

Fairly comprehensive treatment of control-oriented DP is in the two-volume monograph (Bertsekas 2017) and (Bertsekas 2012). It is not available online for free, but the book webpage contains links to other supporting materials including lecture notes. Furthermore, the latest book by the same author (Bertsekas 2023), which is available for free download, contains a decent introduction to dynamic programming.

Having just referenced a book on reinforcement learning (RL), indeed, this popular concept — or at least some of its flavours — is closely related to dynamic programming. In fact, it offers a way to overcome some of the limitations of dynamic programming. In our introductory lecture we are not covering RL, but an interested student can take advantage of availability of high-quality resources such as the the RL-related books and other resources by D. Bertsekas) and another recommendable introduction to RL from control systems perspective (Meyn 2022), which is also available for free download.

The book (Sutton and Barto 2018) often regarded as the bible of RL is nice (and freely available for download) but may be rather difficult to read for a control engineer because of major differences in terminology.

References

Anderson, Brian D. O., and John B. Moore. 2007. Optimal Control: Linear Quadratic Methods. Reprint of the 1989 edition. Dover Publications. http://users.cecs.anu.edu.au/~john/papers/BOOK/B03.PDF.

Bertsekas, Dimitri P. 2012. Dynamic Programming and Optimal Control. 4th ed. Vol. II. Belmont, Massachusetts: Athena Scientific. http://athenasc.com/dpbook.html.

———. 2017. Dynamic Programming and Optimal Control. 4th ed. Vol. I. Belmont, Massachusetts: Athena Scientific. http://athenasc.com/dpbook.html.

———. 2023. A Course in Reinforcement Learning. Belmont, Massachusetts: Athena Scientific. https://web.mit.edu/dimitrib/www/RLCOURSECOMPLETE.pdf.

Kirk, Donald E. 2004. Optimal Control Theory: An Introduction. Reprint of the 1970 edition. Dover Publications.

Lewis, Frank L., Draguna Vrabie, and Vassilis L. Syrmo. 2012. Optimal Control. 3rd ed. John Wiley & Sons. https://lewisgroup.uta.edu/FL%20books/Lewis%20optimal%20control%203rd%20edition%202012.pdf.

Meyn, Sean. 2022. Control Systems and Reinforcement Learning. Cambridge University Press. https://meyn.ece.ufl.edu/control-systems-and-reinforcement-learning/.

Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. 2nd ed. Cambridge, Massachusetts: A Bradford Book. http://incompleteideas.net/book/the-book-2nd.html.