10. Reinforcement Learning and Deep RL

What is Reinforcement Learning?

Reinforcement Learning (RL) is about learning how to act. An agent interacts with an environment, takes actions, receives rewards, and gradually learns a strategy that produces high long-term reward.

Unlike supervised learning, the agent is not given the correct answer for every situation. It must learn from consequences.

Examples:

• A robot learns to walk.
• A game agent learns to win.
• A wireless controller learns channel access or power allocation.
• A vehicle learns safe merging behavior in simulation.

The RL Loop

Every RL problem has a feedback loop:

1. The agent observes the current state.
2. The agent chooses an action.
3. The environment changes.
4. The agent receives a reward.
5. The agent observes the next state.

Key Components

Agent

The learner or decision maker.

Examples: robot, game-playing AI, scheduler, radio resource controller.

Environment

The world the agent interacts with.

Examples: a game, a robot simulation, a wireless network, a traffic system.

State

The information available to the agent at a given time.

Examples: robot joint positions, channel state, queue length, game screen, or vehicle speed.

Action

The choice made by the agent.

Examples: move left, increase transmit power, select a beam, allocate a resource block.

Reward

The feedback signal.

Examples: +1 for success, -1 for collision, high reward for throughput, penalty for energy use.

Policy

The policy is the agent's behavior rule. It maps states to actions.

policy: state -> action

The goal of RL is to learn a policy that maximizes long-term reward.

Immediate Reward vs. Long-Term Return

RL is difficult because the best action now may not give the best immediate reward. The agent must think about future consequences.

Example:

• A robot may take a slow step now to avoid falling later.
• A network scheduler may serve a weaker user now to improve fairness over time.
• A game agent may sacrifice a piece now to win later.

The discounted return is a common way to value future rewards:

• $r_{t+1}$: reward received after action at step $t$.
• $G_t$: discounted future return from step $t$.

A finite episode stops the sum at termination.

• $\gamma$: discount factor.

A larger discount factor gives later rewards greater weight.

Exploration vs. Exploitation

The agent must balance two behaviors:

• Exploration: Try new actions to discover better strategies.
• Exploitation: Use the best-known action to collect reward.

If the agent only exploits, it may get stuck with a mediocre strategy. If it only explores, it may never use what it has learned.

Common exploration methods:

• Epsilon-greedy action selection.
• Softmax action selection.
• Upper confidence bound.
• Entropy bonuses in policy-gradient methods.

Value Functions

A value function estimates how good a state or action is.

State Value

The state value estimates the expected future reward from a state if the agent follows a policy.

$S_t$ is state at step $t$, $\pi$ the followed policy, and $\mathbb E_\pi$ its expectation.

Action Value

The action value estimates the expected future reward after taking action a in state s.

$A_t$ is the action at step $t$. Policy-specific values differ from the optimal action values sought by Q-learning.

Q-values are central to Q-learning and Deep Q-Networks.

Q-Learning

Q-learning is a classic value-based RL algorithm. It learns a table of action values.

For each state-action pair, the agent updates its estimate using the reward received and the estimated value of the next state.

For a transition $(s,a,r,s')$, update:

• $\alpha$: learning rate.
• $\gamma$: discount factor.
• $y$: Q-learning update target.
• $r$: reward for the observed transition.
• $s$: current state.
• $a$: current action.
• $s'$: next state.

At a true terminal state use $y=r$. An external time-limit truncation ends a rollout but can still require bootstrapping for an ongoing task. In the lab, $\varepsilon$ is the probability of drawing a random training action; the displayed greedy route uses the largest learned Q instead.

Intuition

If an action leads to good future rewards, increase its Q-value. If it leads to bad outcomes, decrease its Q-value.

Limitation

Q-learning works well when the state and action spaces are small. It becomes impractical when states are huge, such as raw images or continuous sensor readings.

Policy Gradient Methods

Policy gradient methods learn the policy directly. Their updates increase the probability of actions associated with better return, rather than applying the tabular Q-learning update above.

Intuition

If an action leads to high reward, make that action more likely in similar situations. If it leads to poor reward, make it less likely.

These methods are useful for continuous action spaces, such as robot control or power-control problems.

Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) combines neural networks with RL. The neural network approximates a policy, a value function, or both.

The breakthrough result from DeepMind showed that a Deep Q-Network could learn to play Atari games directly from raw pixels. This connected representation learning with reinforcement learning at scale.

Deep Q-Network

A Deep Q-Network (DQN) replaces the Q-table with a neural network.

Key Ideas

• Experience replay: Store past transitions and sample them randomly during training.
• Target network: Use a slowly updated copy of the network to stabilize targets.
• CNN feature extraction: Learn useful representations from game images or other high-dimensional observations.

These ideas made value-based deep RL much more stable.

Actor-Critic Methods

Actor-critic methods combine two models:

• Actor: Chooses actions.
• Critic: Estimates how good those actions are.

The critic helps the actor learn more efficiently. Many modern DRL methods are actor-critic variants.

RL in Wireless and Communication Systems

RL is attractive in communication systems because many problems involve sequential decisions under uncertainty.

Examples:

• Beam selection.
• Power control.
• Spectrum access.
• Handover decisions.
• Edge computing offloading.
• RIS phase configuration.
• Resource allocation under changing traffic.

However, RL must be used carefully. Real wireless systems have safety, latency, sample-efficiency, and explainability constraints. Simulation-to-reality mismatch is also a major issue.

Practical Challenges

• Sample inefficiency: RL may need many interactions.
• Reward design: A poorly designed reward can produce unwanted behavior.
• Stability: Training can be noisy or unstable.
• Safety: Exploration can be risky in real systems.
• Generalization: A policy trained in one environment may fail in another.
• Evaluation: Results should be tested across many random seeds and scenarios.

Python: Minimal Q-Learning Table

This tiny example shows the skeleton of tabular Q-learning. It uses Gymnasium FrozenLake with deterministic movement. A true terminal transition has no bootstrap; an external truncation ends the rollout while retaining its next-state value.

import numpy as np
import gymnasium as gym

env = gym.make("FrozenLake-v1", is_slippery=False)
env.reset(seed=42)
n_states = env.observation_space.n
n_actions = env.action_space.n
q = np.zeros((n_states, n_actions))

alpha = 0.1
gamma = 0.99
epsilon = 0.1
rng = np.random.default_rng(42)

for episode in range(1000):
    state = env.reset()[0]
    done = False

    while not done:
        if rng.random() < epsilon:
            action = rng.integers(n_actions)
        else:
            action = np.argmax(q[state])

        next_state, reward, terminated, truncated, info = env.step(action)
        done = terminated or truncated

        best_next = 0.0 if terminated else np.max(q[next_state])
        q[state, action] += alpha * (
            reward + gamma * best_next - q[state, action]
        )

        state = next_state

Python: Load a DRL Agent with Stable-Baselines3

For practical DRL, it is better to start with a tested library. Stable-Baselines3 provides implementations of DQN, PPO, A2C, SAC, TD3, and other algorithms.

# pip install stable-baselines3 gymnasium

import gymnasium as gym
from stable_baselines3 import DQN

env = gym.make("CartPole-v1")

model = DQN(
    "MlpPolicy",
    env,
    learning_rate=1e-3,
    buffer_size=50_000,
    learning_starts=1_000,
    batch_size=32,
    gamma=0.99,
    verbose=1,
)

model.learn(total_timesteps=20_000)
model.save("dqn_cartpole")

loaded_model = DQN.load("dqn_cartpole", env=env)
obs, info = env.reset()

for _ in range(500):
    action, _ = loaded_model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()

Python: PPO for Continuous or More Stable Control

PPO is often a strong first choice because it is relatively stable across many environments.

import gymnasium as gym
from stable_baselines3 import PPO

env = gym.make("CartPole-v1")

agent = PPO("MlpPolicy", env, verbose=1)
agent.learn(total_timesteps=20_000)

agent.save("ppo_cartpole")

For wireless-system research, the main work is usually not calling DQN() or PPO(). The hard part is designing the environment: state representation, action space, reward function, constraints, and realistic channel/network dynamics.

Takeaway

• Reinforcement learning: choose actions using feedback from rewards.
• Value-based method: estimate how useful each action is.
• Policy-gradient method: learn action preferences directly.
• Deep RL: uses neural networks for larger state spaces; reward design and reliable evaluation remain essential.

Continue with Transfer Learning to build on this note.