Complete the Q-learning agent skeleton by filling in the missing parts. Each blank is a single keyword, function name, constant name, or integer value.
Each code snippet contains exactly one bug. Type the corrected line in the input box below each one.
Bug 1 — Wrong data structure for Q-table
Bug 2 — Wrong comparison operator in epsilon-greedy
Bug 3 — Wrong sign on death penalty
Complete the update_q() function. You need exactly 4 lines of code:
(1) look up the current Q-value for (state, action),
(2) find the best future Q-value across all actions in next_state,
(3) compute the TD target as reward + GAMMA × best_future,
(4) apply the weighted update to the Q-table.
Add a comment to each line explaining what it does.
Add epsilon decay to the training loop so the agent gradually shifts from exploration to exploitation. Start with EPSILON = 0.5, multiply by 0.995 after each episode, clamp the minimum at 0.01, and print the current epsilon every 500 episodes. Add a comment to each new line.
Test your understanding of Q-learning, the Bellman equation, reward shaping, and epsilon-greedy exploration.
1. What does the Q-table store in a Q-learning agent?
2. In epsilon-greedy, what happens when random.random() < EPSILON is True?
3. In the Bellman equation Q(s,a) ← Q(s,a) + α[r + γ max Q(s′,a′) − Q(s,a)], what role does γ (gamma) play?
4. Why does the snake receive a small negative reward (−0.1) every step, even when nothing special happens?
5. What is the main advantage of defaultdict(float) over a plain {} dict for the Q-table?