🎯Calibrate
1Fill-in
2Debug
3Bellman
4Decay
🧠Concept
📓Reflect
👤 Your Details REQUIRED — fill in before you begin (★ = required)
Please fill in Name, Matric No., and Class before continuing.
🎯 Pre-Activity Calibration Complete this before unlocking the tiers
Before you start, think about what you already know about machine learning and how an agent can learn to play a game. Your prediction helps you track how your understanding grows!
5
Be honest — this is just for your own reflection later!



✏️ Tier 1 Fill in the Blanks — Q-Learning Agent 6 blanks. All must be correct to advance.

Complete the Q-learning agent skeleton by filling in the missing parts. Each blank is a single keyword, function name, constant name, or integer value.

from collections import ___1___(float) q_table = defaultdict(float) ___2___ = 0.1 # exploration rate (10% random) ALPHA = 0.2 # learning rate GAMMA = 0.9 # discount factor def choose_action(state): if random.random() < EPSILON: # explore return random.choice(ACTIONS) return ___3___(q_table[(state, a)] # exploit for a in ACTIONS) def reward(ate_food, died): if died: return -___4___ if ate_food: return ___4___ return -0.1 def update_q(state, action, r, next_state): old_q = q_table[(state, action)] best = max(q_table[(next_state, a)] for a in ACTIONS) q_table[(state, action)] = old_q + ___5___ * (r + ___6___ * best - old_q)
1from collections import ___1___(float) — a collections class that auto-creates 0.0 for missing keys
2___2___ = 0.1 — constant name for the exploration rate used in choose_action
3return ___3___(...) — built-in Python function that returns the highest value in an iterable
4return ±___4___ — the integer magnitude of both the food reward and the death penalty
5old_q + ___5___ * (...) — the learning-rate constant that controls how fast Q-values are updated
6r + ___6___ * best — the discount-factor constant that weighs future rewards vs immediate ones

🐛 Tier 2 Debug the Q-Learning Code 3 bugs hidden below. Fix each one. 2 attempts max.

Each code snippet contains exactly one bug. Type the corrected line in the input box below each one.

Bug 1 — Wrong data structure for Q-table

q_table = {} # Q-table initialisation
Hint: A plain dict raises KeyError for any unseen state-action pair. The Q-table needs to return 0.0 automatically for unknown entries.

Bug 2 — Wrong comparison operator in epsilon-greedy

if random.random() > EPSILON: # inside choose_action
Hint: Epsilon is the probability of exploring (choosing randomly). The condition should be true when a random number is less than epsilon.

Bug 3 — Wrong sign on death penalty

if died: return 10 # inside reward()
Hint: Dying is bad! The agent should receive a large negative reward when it hits a wall or itself.
REFLECTION Mid-Activity Reflection


🔧 Tier 3 Bellman — Write the Full Q-Update Implement update_q() step by step using the Bellman equation.

Complete the update_q() function. You need exactly 4 lines of code: (1) look up the current Q-value for (state, action), (2) find the best future Q-value across all actions in next_state, (3) compute the TD target as reward + GAMMA × best_future, (4) apply the weighted update to the Q-table. Add a comment to each line explaining what it does.

from collections import defaultdict ALPHA = 0.2 # learning rate GAMMA = 0.9 # discount factor ACTIONS = ['UP', 'DOWN', 'LEFT', 'RIGHT'] q_table = defaultdict(float) def update_q(state, action, reward, next_state): # Write your 4-line Bellman update here (with comments). pass


🏆 Tier 4 Extend — Add Epsilon Decay Make the agent explore less as it learns more.

Add epsilon decay to the training loop so the agent gradually shifts from exploration to exploitation. Start with EPSILON = 0.5, multiply by 0.995 after each episode, clamp the minimum at 0.01, and print the current epsilon every 500 episodes. Add a comment to each new line.

NUM_EPISODES = 5000 EPSILON = 0.5 # start exploring 50% of the time for episode in range(NUM_EPISODES): run_episode() # --- Add your epsilon decay code below this line --- # --- End of your addition --- if episode % 500 == 0: print(f"Episode {episode}: epsilon = {EPSILON:.4f}")


🧠 Concept Concept Check — Q-Learning & Reinforcement Learning 5 multiple-choice questions. Answer all then submit.

Test your understanding of Q-learning, the Bellman equation, reward shaping, and epsilon-greedy exploration.

1. What does the Q-table store in a Q-learning agent?

2. In epsilon-greedy, what happens when random.random() < EPSILON is True?

3. In the Bellman equation Q(s,a) ← Q(s,a) + α[r + γ max Q(s′,a′) − Q(s,a)], what role does γ (gamma) play?

4. Why does the snake receive a small negative reward (−0.1) every step, even when nothing special happens?

5. What is the main advantage of defaultdict(float) over a plain {} dict for the Q-table?

POST-ACTIVITY Post-Activity Calibration & Reflection
5
Your self-estimate — be honest!



JOURNAL Reflective Journal — Step 10 (min 5 words per prompt)