Explanation for tile3 (ChatGPT)

This is Rich Sutton’s tile coding implementation for reinforcement learning. It converts continuous state variables into a small set of active discrete feature indices.

Main idea

Tile coding represents a continuous point, such as:

floats = [x_position, velocity]

by placing several overlapping grids over the space. Each grid is called a tiling. For each tiling, the point activates exactly one tile. So if there are 8 tilings, the output is 8 active tile indices.

This is useful because instead of using the raw continuous state, RL algorithms can use a sparse binary feature vector.

For example:

tiles(iht, 8, [1.2, 3.7])

returns something like:

[0, 1, 2, 3, 4, 5, 6, 7]

Each number is the index of an active feature.

IHT: Index Hash Table

class IHT:

IHT stores a mapping from tile coordinates to integer feature indices.

For example, internally it may map:

(0, 3, 5) -> 17
(1, 3, 6) -> 18

The constructor:

IHT(sizeval)

creates a table with a maximum size.

Example:

iht = IHT(4096)

This means there can be up to 4096 unique tile indices.

getindex

def getindex(self, obj, readonly=False):

This looks up a coordinate tuple.

If the coordinate was seen before, it returns its existing index.

If it is new and the table is not full, it assigns the next available index.

If the table is full, it starts using hash collisions:

return basehash(obj) % self.size

So after the table fills up, different coordinates may map to the same index.

hashcoords

def hashcoords(coordinates, m, readonly=False):

This converts tile coordinates into feature indices.

It behaves differently depending on m:

hashcoords(coords, iht)

uses the IHT table.

hashcoords(coords, 4096)

directly hashes coordinates into indices from 0 to 4095.

hashcoords(coords, None)

returns the raw coordinates without hashing. This is useful for debugging.

tiles

def tiles(ihtORsize, numtilings, floats, ints=[], readonly=False):

This is the main function.

Inputs:

ihtORsize

is either an IHT, an integer table size, or None.

numtilings

is the number of overlapping grids.

floats

are the continuous variables.

ints

are optional discrete variables, such as an action.

For example:

tiles(iht, 8, [position, velocity], [action])

would create tile features for a state-action pair.

How tiles works

First, continuous values are scaled into integer coordinates:

qfloats = [floor(f*numtilings) for f in floats]

So if:

floats = [1.25]
numtilings = 8

then:

qfloats = [floor(10.0)] = [10]

Then the code loops over each tiling:

for tiling in range(numtilings):

Each tiling gets slightly different offsets, so the grids do not all line up.

The coordinate list starts with the tiling number:

coords = [tiling]

This ensures that tile (3, 5) in tiling 0 is different from tile (3, 5) in tiling 1.

Then each continuous variable is assigned to a tile coordinate:

coords.append((q + b) // numtilings)

The variable b creates the offset between tilings.

Finally, integer variables are added:

coords.extend(ints)

Then the coordinates are converted to an index:

Tiles.append(hashcoords(coords, ihtORsize, readonly))

The final result is a list of active tile indices.

Example

iht = IHT(4096)

active_tiles = tiles(iht, 8, [1.2, 3.4])

This returns 8 tile indices, because there are 8 tilings:

[0, 1, 2, 3, 4, 5, 6, 7]

Later, a nearby point might return many of the same indices:

tiles(iht, 8, [1.21, 3.39])

This is the key idea: nearby states share features, so learning generalizes locally.

tileswrap

def tileswrap(ihtORsize, numtilings, floats, wrapwidths, ints=[], readonly=False):

This version is similar to tiles, but supports wrapping dimensions.

This is useful for cyclic variables, such as angles.

For example, an angle near 0 should be close to an angle near 2π.

The wrapping happens here:

coords.append(c % width if width else c)

If a wrap width is provided, the coordinate wraps around modulo that width.

Example:

tileswrap(iht, 8, [angle], [10])

means the tile coordinate wraps every 10 units.

Why this is used in reinforcement learning

Tile coding creates features like this:

feature_vector[tile_index] = 1

All other entries are zero.

So instead of storing values for exact continuous states, we approximate:

V(s) = wᵀ x(s)

or:

Q(s, a) = wᵀ x(s, a)

where x(s) is the sparse tile-coded feature vector.

Only the weights for the active tiles are updated.

For example:

for tile in active_tiles:
    weights[tile] += alpha * error

Since nearby states share some tiles, updates generalize to nearby states.

Important detail

The input floats must be scaled before calling tiles.

The code assumes tile width is about 1 in each dimension. So if your original state variable ranges from [-1.2, 0.6], you usually scale it first:

scaled_position = position / position_tile_width

Then pass the scaled values into tiles.

In short: this code maps continuous states into a fixed number of discrete active features using several offset grids. It is a compact way to make linear reinforcement learning methods work with continuous state spaces.