KullGames GridWorldAI
Tabular Q-learning (Sutton & Barto S6.5) learns an optimal policy on a 6x6 ASCII gridworld purely from trial and error. Training and the greedy evaluation below are run live at build time by OnlyCSharp 3.0 ArtificialIntelligence.QLearning (QLearningAgent + QLearningTrainer + GridWorld).
The map (S=start, G=goal, #=wall, X=hazard, .=floor)
| S | . | # | . | . | . |
| . | . | # | . | # | . |
| . | . | # | . | # | . |
| . | . | . | . | # | . |
| . | # | # | # | X | . |
| . | . | . | . | # | G |
Training (8000 episodes, alpha=0.2, gamma=0.95, epsilon 1->0.05)
| episode bucket | avg return |
|---|---|
| 0 - 499 | -7.231 |
| 500 - 999 | -5.767 |
| 1000 - 1499 | -4.025 |
| 1500 - 1999 | -2.813 |
| 2000 - 2499 | -2.132 |
| 2500 - 2999 | -1.472 |
| 3000 - 3499 | -1.050 |
| 3500 - 3999 | -0.833 |
| 4000 - 4499 | -0.555 |
| 4500 - 4999 | -0.376 |
| 5000 - 5499 | -0.303 |
| 5500 - 5999 | -0.114 |
| 6000 - 6499 | 0.020 |
| 6500 - 6999 | 0.109 |
| 7000 - 7499 | 0.233 |
| 7500 - 7999 | 0.325 |
Goal reached in 5984/8000 training episodes (74.8%); final epsilon=0.050.
Learned greedy policy - path taken
Greedy rollout: 16 steps, total reward=0.400, reached terminal=True.
| step | from | action | to |
|---|---|---|---|
| 1 | (0,0) | East | (1,0) |
| 2 | (1,0) | South | (1,1) |
| 3 | (1,1) | South | (1,2) |
| 4 | (1,2) | South | (1,3) |
| 5 | (1,3) | East | (2,3) |
| 6 | (2,3) | East | (3,3) |
| 7 | (3,3) | North | (3,2) |
| 8 | (3,2) | North | (3,1) |
| 9 | (3,1) | North | (3,0) |
| 10 | (3,0) | East | (4,0) |
| 11 | (4,0) | East | (5,0) |
| 12 | (5,0) | South | (5,1) |
| 13 | (5,1) | South | (5,2) |
| 14 | (5,2) | South | (5,3) |
| 15 | (5,3) | South | (5,4) |
| 16 | (5,4) | South | (5,5) |
Path overlay (o = a floor cell the greedy policy walked through)
| S | o | # | o | o | o |
| . | o | # | o | # | o |
| . | o | # | o | # | o |
| . | o | o | o | # | o |
| . | # | # | # | X | o |
| . | . | . | . | # | G |
Verdict
SUCCESS: the trained agent reaches the goal G in 16 steps (optimal is 16), avoiding hazard X.