Course Description

Reinforcement Learning (RL) is the study of sequential decision making under uncertainty with applications abounding a vast range of real-world problems, from advertisements and recommendations systems to robotics and autonomous vehicles. In this course, we learn how to approach RL problems from both theoretical and empirical viewpoints. We study how RL agents interact with their surrounding environments and learn how to improve their behavior over time. After this course, we will be comfortable understanding how companies like Amazon, Yelp, Netflix, Adobe, and Google provide their recommendation services, and how companies like DeepMind can master video games such as Atari games and StarCraft.

The course assumes students are comfortable with basic concepts in analysis, elementary algebra, probability theory, statistics, and programming. It is recommended that students take courses in linear algebra, measure theory, learning theory, and deep learning, but it is not necessary. We will cover the required material in this course.

Course Details

• Lectures on M/W at 4:30pm-5:45pm in Lawson B155
• Office hours, one hour right after class, M/W at 5:45pm-6:45pm

  *Make sure to enroll in the course piazza(passcode in Brightspace).

  *Lecture videos will be streamed and uploaded to Brightspace.

  *For information regarding auditing and enrolling contact Lori O’Brien.

  *We do not use a particular textbook.

Grading

• Homework Assignments

  [HW1: 8%, HW2: 8%, HW3[midterm]: 20%, HW4: 8%, HW5: 8%, HW6: 8%, HW7[final]: 20%]

• Project(Individual project/no group submission) 20%. Deadline Dec 8th 2020.

*No late submission would be accepted.

Collaboration Policy

I encourage the students to discuss and exchange ideas for the problem sets and the project, but the students shall accomplish each of their problem sets and project separately. Make sure you do not violate the Honor Code. If you are discussing with other students but cannot be informative enough without consulting your solution, please do not pursue the discussion. Instead, refer your colleagues and classmates to a teaching assistant.

Topics

• Probability Theory, including basic concepts and concentration inequalities
• Bandits, including bandit algorithms in Multi-Armed Bandits, and other variants
• Markov Decision Processes (MDP), including basic concepts of optimization and dynamic programming
• RL algorithms in MDPs, including value based methods and policy based methods
• Control Theory, including linear dynamical systems
• Deep Reinforcement Learning, including deep Q-Networks, and policy gradient methods

Course lecture notes

• [Lecture notes]

Extra Resources

• Book: Probability with Martingales by David Williams
• Book: Real and Complex Analysis by Walter Rudin
• Book: High-Dimensional Statistics: A Non-Asymptotic Viewpoint by Martin Wainwright
• Book: High-Dimensional Probability: An Introduction with Applications in Data Science by Roman Vershynin
• Book: Bandit Algorithms by Tor Lattimore and Csaba Szepesvári
• Book: Markov Decision Processes: Discrete Stochastic Dynamic Programming by Martin Puterman
• Book: Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto
• Book: Algorithms for Reinforcement Learning by Csaba Szepesvárii
• Book: Reinforcement Learning: Theory and Algorithms by Alekh Agarwal, Nan Jiang, and Sham M. Kakade
• Book: Dynamic Programming and Optimal Control by Dimitri Bertsekas
• [Course on RL] by Csaba Szepesvári
• [Course on RL] by David Silver
• [Introduction to Artificial Intelligence] by Pieter Abbeel
• [DeepRL] by Sergey Levine
• [Course on RL] by Pierre-Luc Bacon
• [Course on RL],[Course on RL] by Nan Jiang

Note-Covid-19

In case you observe symptoms or become quarantined or isolated at any point in time during the semester, immediately reach out to the Protect Purdue Health Center, Academic Case Manager, and the Office of the Dean of Students.