Reinforcement Learning: Agents, Environments, and Rewards in Practice

Scenario context

Reinforcement Learning trains agents to make sequences of decisions by learning from rewards and penalties. Unlike supervised learning, RL learns through trial and error rather than labeled examples. Use it for sequential decision problems where the optimal action depends on future consequences — robotics, game AI, recommendation systems, and RLHF training for LLMs.

You're training a robot arm, but there's no…

This walkthrough focuses on The Trial-and-Error Learning Problem: When Labels Don't Exist.

Before making a design choice, clarify:

• You're training a robot arm, but there's no labeled dataset.
• It needs to figure out through trial and error how to grip objects without breaking them, learning from thousands of attempts where it either succeeds or drops the object.
• Traditional supervised learning can't solve this because there's no "correct grip pressure" label for each object — the robot must discover the optimal policy through exploration.

Concepts in play: deep-learning-foundations, llm-agents-and-langgraph, quantization-and-llm-inference, database-indexing

Stage 1 — RL Components: The Agent-Environment Feedback Loop

• Reinforcement Learning involves four core components interacting in a continuous loop: Agent : The learner/decision-maker (e.g., chess AI, trading algorithm, robot controller) Environment : The external system the agent…
• Agent observes the current state s t 2.
• Agent selects an action a t based on its policy π(a s) 3.
• Environment transitions to new state s {t+1} and returns reward r t 4.

graph TD
    A[Agent] -->|Action| E[Environment]
    E -->|State, Reward| A
    E -->|Next State| E

    subgraph "RL Components"
        A
        E
        S[State Space]
        R[Reward Function]
        P[Policy ]
    end

    style A fill:#e1f5fe
    style E fill:#f3e5f5
    style S fill:#e8f5e8
    style R fill:#fff3e0
    style P fill:#fce4ec

Observable outcome: Environment transitions to new state s {t+1} and returns reward r t 4.

Stage 2 — How Q-Learning Discovers Optimal Policies Through Trial and Error

• The core challenge in RL is the credit assignment problem : if the agent wins a game after 50 moves, which moves were actually good?
• Q-Learning solves this by learning the Q-function — the expected reward for taking action a in state s and following the optimal policy afterward.
• Initialize Q-table Q(s,a) with small random values 2.
• For each episode: - Start in initial state s - While not terminal: - Choose action a using ε-greedy policy - Execute action, observe reward r and next state s' - Update: Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]…

Q(s,a) = r + γ max Q(s',a')

Observable outcome: For each episode: - Start in initial state s - While not terminal: - Choose action a using ε-greedy policy - Execute action, observe reward r and next state s' - Update: Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]…

Stage 3 — Deep Dive: How Modern RL Algorithms Scale Beyond Tabular Methods

• The Internals Tabular Q-Learning stores Q-values in a lookup table, but this breaks down with large state spaces.
• A chess game has ~10^43 possible positions — impossible to store in memory.
• Deep Q-Networks (DQN) replace the Q-table with a neural network that approximates Q(s,a).
• The network takes state s as input and outputs Q-values for all possible actions: Experience Replay : DQN stores (state, action, reward, next state) transitions in a replay buffer and samples random batches for training.

# DQN Architecture
state_input = tf.keras.Input(shape=state_size)
x = tf.keras.layers.Dense(64, activation='relu')(state_input)
x = tf.keras.layers.Dense(64, activation='relu')(x)
q_values = tf.keras.layers.Dense(action_size, activation='linear')(x)
model = tf.keras.Model(state_input, q_values)

# REINFORCE update rule
policy_loss = -tf.reduce_mean(log_probs * returns)
optimizer.minimize(policy_loss)

Observable outcome: The network takes state s as input and outputs Q-values for all possible actions: Experience Replay : DQN stores (state, action, reward, next state) transitions in a replay buffer and samples random batches for training.

Stage 4 — Visualizing the RL Learning Process

• This diagram shows how modern RL agents learn: 1.
• Agent takes action in environment 2.
• Environment returns new state and reward 3.
• Experience stored in replay buffer 4.

sequenceDiagram
    participant A as Agent
    participant E as Environment
    participant M as Memory/Replay Buffer
    participant N as Neural Network

    loop Training Episode
        A->>E: action_t
        E->>A: state_{t+1}, reward_t
        A->>M: store (s_t, a_t, r_t, s_{t+1})

        Note over A: Sample mini-batch from M
        A->>N: Update Q-network
        N->>A: Updated Q-values

        Note over A: -greedy action selection
    end

    Note over A,N: Policy improves over time

Observable outcome: Experience stored in replay buffer 4.

Stage 5 — Real-World Applications: From Atari to RLHF

• Case Study 1: Game AI Mastery AlphaGo/AlphaZero used RL to master Go, chess, and shogi without human game data.
• The agent learned by playing millions of games against itself, starting with random moves and gradually discovering winning strategies.
• Input : Board position (19×19 grid for Go) Process : Monte Carlo Tree Search guided by neural network value estimates Output : Move selection that maximizes win probability Scaling insight : Self-play generates infinite…
• Case Study 2: LLM RLHF Training Reinforcement Learning from Human Feedback (RLHF) trains language models to be helpful and harmless.

Observable outcome: Case Study 2: LLM RLHF Training Reinforcement Learning from Human Feedback (RLHF) trains language models to be helpful and harmless.

Stage 6 — Gymnasium: The OpenAI Standard for RL Environments

• Gymnasium (formerly OpenAI Gym) provides standardized environments for RL research and development.
• It defines a common interface that works with any RL algorithm: Why Gymnasium Matters : It standardizes the RL interface across different environments — from simple control tasks like CartPole to complex simulations lik…
• The same algorithm can work on any Gymnasium-compatible environment with minimal code changes.
• Environment Categories : - Classic Control : CartPole, MountainCar (simple physics) - Atari : Video games with pixel observations - MuJoCo : Continuous control robotics simulation - Custom : Easy to wrap any environment…

import gymnasium as gym
import numpy as np
from collections import defaultdict

# Create CartPole environment
env = gym.make("CartPole-v1", render_mode="human")

# Simple Q-learning for discrete state approximation
class QLearningAgent:
    def __init__(self, action_size, learning_rate=0.1, discount=0.95, epsilon=0.1):
        self.q_table = defaultdict(lambda: np.zeros(action_size))
        self.lr = learning_rate
        self.gamma = discount
        self.epsilon = epsilon

    def discretize_state(self, state):
        """Convert continuous state to discrete bins for Q-table"""
        # CartPole state: [position, velocity, angle, angular_velocity]
        bins = [
            np.digitize(state[0], np.linspace(-2.4, 2.4, 10)),
            np.digitize(state[1], np.linspace(-2, 2, 10)), 
            np.digitize(state[2], np.linspace(-0.2, 0.2, 10)),
            np.digitize(state[3], np.linspace(-2, 2, 10))
        ]
        return tuple(bins)

    def choose_action(self, state):
        """ε-greedy action selection"""
        discrete_state = self.discretize_state(state)
        if np.random.random() < self.epsilon:
            return env.action_space.sample()  # Random action
        return np.argmax(self.q_table[discrete_state])  # Best known action

    def update(self, state, action, reward, next_state, done):
        """Q-learning update rule"""
        discrete_state = self.discretize_state(state)
        discrete_next_state = self.discretize_state(next_state)

        # Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]
        current_q = self.q_table[discrete_state][action]
        if done:
            target_q = reward
        else:
            target_q = reward + self.gamma * np.max(self.q_table[discrete_next_state])

        self.q_table[discrete_state][action] += self.lr * (target_q - current_q)

# Training loop
agent = QLearningAgent(action_size=env.action_space.n)
episodes = 1000
scores = []

for episode in range(episodes):
    state, info = env.reset()
    total_reward = 0

    while True:
        action = agent.choose_action(state)
        next_state, reward, terminated, truncated, info = env.step(action)

        # Custom reward shaping for CartPole
        # Reward staying upright, penalize extreme positions
        angle_reward = 1.0 - abs(next_state[2]) / 0.2  # Angle penalty
        position_reward = 1.0 - abs(next_state[0]) / 2.4  # Position penalty
        shaped_reward = reward + 0.1 * (angle_reward + position_reward)

        agent.update(state, action, shaped_reward, next_state, terminated or truncated)
        state = next_state
        total_reward += reward

        if terminated or truncated:
            break

    scores.append(total_reward)

    # Decay exploration rate
    if agent.epsilon > 0.01:
        agent.epsilon *= 0.995

    if episode % 100 == 0:
        avg_score = np.mean(scores[-100:])
        print(f"Episode {episode}, Average Score: {avg_score:.2f}, ε: {agent.epsilon:.3f}")

env.close()

Observable outcome: Environment Categories : - Classic Control : CartPole, MountainCar (simple physics) - Atari : Video games with pixel observations - MuJoCo : Continuous control robotics simulation - Custom : Easy to wrap any environment…

End-to-end outcome

After working through the stages, you should be able to explain:

• • RL learns through trial and error : Agents discover optimal policies by exploring environments and learning from reward feedback, unlike supervised learning which requires labeled examples - Sequential decision-making…

Alternatives considered

Decision Guide: When to Choose RL Over Supervised Learning

Rule of thumb : Use RL when the optimal action depends on future consequences, not just current context.

Failure and degraded modes

Performance vs Sample Efficiency

• Detection: RL typically requires orders of magnitude more data than supervised learning.
• User or operator impact: RL typically requires orders of magnitude more data than supervised learning.

Exploration vs Exploitation Dilemma

• Detection: Pure exploitation (always choose best known action) gets stuck in local optima.
• User or operator impact: Pure exploitation (always choose best known action) gets stuck in local optima.

Common Failure Modes

• Detection: Reward Hacking : Agent finds unexpected ways to maximize reward that don't align with intended behavior.
• User or operator impact: Reward Hacking : Agent finds unexpected ways to maximize reward that don't align with intended behavior.

Lessons Learned: Common Pitfalls and Best Practices

• Detection: Reward Engineering Is Critical : The reward function defines what the agent optimizes for.
• User or operator impact: Reward Engineering Is Critical : The reward function defines what the agent optimizes for.