For my thesis I built an agent that learns where food is from nothing but its own movement, then moves by climbing that value, with no map and no path planner. I expected its neurons to look like the cells homing pigeons use to find food. One of three behaviours came through, and the two misses taught me more.
What I did, for my thesis
- QuestionCan a value network grow the cells pigeons use to find food?
- Reward designTwo schedules: steady food, then food that runs out
- State checkA regression from the state back to position
- Value networkTD learning in TensorFlow
- NavigationThe Go, Explore, NoGo rule, then a firing map for every unit
2019 to 2020 · Dual degree thesis, Aerospace Engineering, IIT Madras · Guided by Prof. S. R. Chakravarthy and Prof. V. Srinivasa Chakravarthy
As built: it wanders
Trigger sized to a step: it heads for food
Go keeps going, Explore wanders, NoGo turns back. The number counts pushes toward food.
Research
Did it behave like a pigeon?
Drones find their way with GPS and hand written rules. Pigeons forage and home without either. Recordings from pigeons show three behaviours, so I checked the agent for each.
Food cells✓ Reproduced
PigeonCells fire at the food bowls
Agent70 to 75% of its neurons fired at the bowls
Route cells✕ Not reproduced
PigeonCells fire on the route, then go quiet
AgentNone of its neurons fired on the route
A visiting order✕ Not reproduced
PigeonVisits the bowls clockwise or anticlockwise
AgentNo order it went from bowl to bowl at random
System design
How it works
It learns how good each place is from surprise, then climbs: keep going while value rises, turn back when it falls, wander in between.
Judgement
Six decisions
Paid off
Cost me later
Research
What emerged
1.8×the value for twice the food
It learned where food is and how much. But look before each visit: value stays at 0 until the food lands. It never saw the food coming.



Most neurons became food cells
Each one shown here fires in a small patch at one bowl, like the location cells found in pigeons.

Looking back
What held it back, and the fix
0.11is what one step changed value by
That holds for 99% of steps. The agent needed a jump of 0.7 to act, so only food arriving or running out ever cleared it. Everywhere in between, it wandered.
Cause
Value copied the foodReward was an input, so value sat at 0 until the food arrived, then jumped to 1.0 or 1.8.Fix
Feed reward only through the learning signalValue then has to predict food, not echo it.Cause
It sensed food 2 steps aheadWith a discount of 0.5, value was near zero over half to two thirds of each maze.Fix
Raise the discount to about 0.9About 10 steps of look ahead, so value rises on the way to a bowl.Cause
Steps never cleared the triggerOne step changed value by under 0.11 in 99% of steps. Acting needed 0.7.Fix
Size the trigger to a single stepAnd fix a bug that cancelled Go. The lower replay shows the rule waking up.Cause
No route cells, no orderA critic rates places. Nothing in it chooses a route.Fix
Add an actor, with heading in the stateSomething that learns routes is where route cells and an order can come from.
After
Where it led
After the thesis I joined the same lab as a research engineer. The question moved from value to space itself.