Chapters 9-10: Prediction and Control using Approximation

Introduction

• Explanation: What is the on-Policy State Distribution
• Example: Implementation of state-action features Semi-gradient TD(0), Semi-gradient Sarsa. Prediction and Control with approximation using linear features for a simple grid world (no walls).

Examples Where Simple Linear Features Fail

• Linear approximation with linear features for the 4x3 Grid World.
• Linear approximation with linear features for the L-Maze.

Advanced Feature Construction

• Linear approximation with Fourier basis features (4x3 Gridworld).
• Linear approximation with Fourier basis features (L-Maze).

Exercise

• Value Function Approximation for the Lunar Lander Problem using Tile Coding

License

© 2026 Michael Hahsler. All code and documents in this repository are provided under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) License.