Skip to content
  1. Projects

RL navigation agent

An agent that learns where food is from its own movement, whose neurons come to fire near the food like cells in homing pigeons.

RL navigation agent
  • TensorFlow
  • TD learning
  • NumPy
  • scikit-learn
1 of 3
pigeon behaviours reproduced
70 to 75%
of neurons became food cells
1.8×
the value for twice the food
2 steps
how far ahead it could sense food

For my thesis I built an agent that learns where food is from nothing but its own movement, then moves by climbing that value, with no map and no path planner. I expected its neurons to look like the cells homing pigeons use to find food. One of three behaviours came through, and the two misses taught me more.

What I did, for my thesis

  1. QuestionCan a value network grow the cells pigeons use to find food?
  2. Reward designTwo schedules: steady food, then food that runs out
  3. State checkA regression from the state back to position
  4. Value networkTD learning in TensorFlow
  5. NavigationThe Go, Explore, NoGo rule, then a firing map for every unit

2019 to 2020 · Dual degree thesis, Aerospace Engineering, IIT Madras · Guided by Prof. S. R. Chakravarthy and Prof. V. Srinivasa Chakravarthy

As built: it wanders

Trigger sized to a step: it heads for food

Go keeps going, Explore wanders, NoGo turns back. The number counts pushes toward food.

Same brain, same maze. The only difference is how big a jump in value it takes to act.

Research

Did it behave like a pigeon?

Drones find their way with GPS and hand written rules. Pigeons forage and home without either. Recordings from pigeons show three behaviours, so I checked the agent for each.

  • Food cells✓ Reproduced

    PigeonCells fire at the food bowls

    Agent70 to 75% of its neurons fired at the bowls

  • Route cells✕ Not reproduced

    PigeonCells fire on the route, then go quiet

    AgentNone of its neurons fired on the route

  • A visiting order✕ Not reproduced

    PigeonVisits the bowls clockwise or anticlockwise

    AgentNo order it went from bowl to bowl at random

System design

How it works

It learns how good each place is from surprise, then climbs: keep going while value rises, turn back when it falls, wander in between.

Architecture of the agent Five stages in a loop through the maze. Sense: speed and heading, 2 signals, the only input. Encode: the lab's model, which I did not write, turns them into 48 numbers through head direction cells, oscillators and two anti Hebbian layers. State: those 48 plus the reward make 49 numbers, and position must be recoverable from them with R squared above 0.9. Value: a critic network, 49 inputs, 200 neurons and 1 output, trained with TD learning and a discount of 0.5. Move: Go keeps going, Explore wanders, NoGo turns back, acting on a jump in value of 0.7. Each step moves the agent in the maze, which changes its speed and heading. 1 · SENSE Speed and heading What an animal feels as it moves. No position, no map. 2 signals in 2 · ENCODE The lab's model Head direction cells drive oscillators, then two anti Hebbian layers 48 numbers out, not my code 3 · STATE 49 numbers 48 from the model, plus the reward Checked first: can it still tell where the agent is? R² > 0.9 or data was redone 4 · VALUE Critic network 49 200 1 Rates each place TD learning, discount 0.5 5 · MOVE Climb value Go keeps going Explore wanders NoGo turns back acts on a 0.7 jump 2 48 49 1 one step speed, heading THE MAZEeach step moves the agent, which changes its speed and heading
Architecture of the agent Five stages in a loop through the maze. Sense: speed and heading, 2 signals, the only input. Encode: the lab's model, which I did not write, turns them into 48 numbers. State: those 48 plus the reward make 49 numbers, and position must be recoverable from them with R squared above 0.9. Value: a critic network, 49 inputs, 200 neurons and 1 output, trained with TD learning and a discount of 0.5. Move: Go keeps going, Explore wanders, NoGo turns back, acting on a jump in value of 0.7. Each step moves the agent in the maze, which changes its speed and heading. 1 · SENSE Speed and heading The only input. No position, no map. 2 signals 2 · ENCODE The lab's model Head direction cells, oscillators, two anti Hebbian layers. Built in the lab, not my code. 48 numbers 3 · STATE 49 numbers 48 from the model, plus the reward R² > 0.9 position still recoverable 49 numbers 4 · VALUE Critic network 49 in · 200 neurons · 1 out Rates each place. TD learning, discount 0.5 1 value 5 · MOVE Climb the value Go keeps going Explore wanders NoGo turns back Acts only on a jump in value of 0.7 one step THE MAZE Each step changes speed and heading

Judgement

Six decisions

Paid off

Cost me later

Movement onlyNo coordinates in, only speed and heading, like an animal.Location was always in the state.
Reward inside the stateValue could read the reward straight from its input.Value copied the food instead of predicting it.
A discount of 0.5Each step ahead counts half, so value stays close to the reward in hand.It sensed food only about 2 steps ahead.
Food that runs outReward stops partway through a visit (55 s of 80 s), as a bowl of pellets would.Value fell the moment the food did.
Value before an actorA critic alone keeps the neurons easy to read.Nothing could learn routes, so no route cells.
A fixed trigger of 0.7The Go, Explore, NoGo rule from models of the basal ganglia, with a set threshold.No single step was big enough to clear it.

Research

What emerged

1.8×the value for twice the food

It learned where food is and how much. But look before each visit: value stays at 0 until the food lands. It never saw the food coming.

Left, a plus maze shaded by learned value: the two bowls with food 1 are dark, the two with food 0.5 lighter. Right, value and food over time: each time food arrives, value jumps from 0 to about 1 or 1.8, and drops back when the food stops. Before each visit it sits flat at 0.
A plus maze shaded by learned value: the two bowls with food 1 are dark, the two with food 0.5 lighter.
Value and food over time, eight visits: each time food arrives, value jumps from 0 to about 1 or 1.8, and drops back when the food stops. Before each visit it sits flat at 0.

Most neurons became food cells

Each one shown here fires in a small patch at one bowl, like the location cells found in pigeons.

Eight small firing maps, four in the plus maze and four in the box. Each shows one compact patch of activity at a single bowl.

Looking back

What held it back, and the fix

0.11is what one step changed value by

That holds for 99% of steps. The agent needed a jump of 0.7 to act, so only food arriving or running out ever cleared it. Everywhere in between, it wandered.

  1. Cause

    Value copied the foodReward was an input, so value sat at 0 until the food arrived, then jumped to 1.0 or 1.8.

    Fix

    Feed reward only through the learning signalValue then has to predict food, not echo it.
  2. Cause

    It sensed food 2 steps aheadWith a discount of 0.5, value was near zero over half to two thirds of each maze.

    Fix

    Raise the discount to about 0.9About 10 steps of look ahead, so value rises on the way to a bowl.
  3. Cause

    Steps never cleared the triggerOne step changed value by under 0.11 in 99% of steps. Acting needed 0.7.

    Fix

    Size the trigger to a single stepAnd fix a bug that cancelled Go. The lower replay shows the rule waking up.
  4. Cause

    No route cells, no orderA critic rates places. Nothing in it chooses a route.

    Fix

    Add an actor, with heading in the stateSomething that learns routes is where route cells and an order can come from.

After

Where it led

After the thesis I joined the same lab as a research engineer. The question moved from value to space itself.