3. Parameters, Training and Generalization

Why These Parameters Matter

Machine learning is not only about choosing an algorithm. The final result also depends on many settings, choices, and training conditions.

Some of these are learned by the model:

• Weights.
• Bias terms.
• Neural-network layer parameters.
• Q-values or policy parameters in reinforcement learning.

Some are chosen by the researcher or engineer before training:

• Learning rate.
• Batch size.
• Number of epochs.
• Random seed.
• Model depth.
• Regularization strength.
• Discount factor in reinforcement learning.

These user-chosen settings are called hyperparameters. They strongly affect speed, stability, accuracy, generalization, and reproducibility.

References for this section3

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Parameter vs. Hyperparameter

Term	Meaning	Example
Parameter	Learned from data during training	Weight w, coefficient β, neural-network bias
Hyperparameter	Chosen before or during training setup	Learning rate, batch size, number of layers
Metric	Used to evaluate performance	Accuracy, F1-score, MAE, reward
Loss	Function the model tries to minimize	Cross-entropy, MSE

Simple rule:

If the model learns it, it is usually a parameter.
If you set it before training, it is usually a hyperparameter.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Random Seed

A seed controls randomness so experiments can be repeated.

Randomness appears in:

• Train-test splitting.
• Weight initialization.
• Data shuffling.
• Dropout.
• Mini-batch selection.
• Reinforcement-learning exploration.

If two experiments use the same seed and the same software/hardware behavior, the results are more likely to match.

import random
import numpy as np

random.seed(42)
np.random.seed(42)

References for this section3

• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Effect of Seed

Seed behavior	Effect
Fixed seed	Better reproducibility
Different seeds	Better estimate of result stability
Reporting only one lucky seed	Can exaggerate performance

Good practice is to run important experiments with multiple seeds and report the mean and variation.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Dataset Split

A dataset is usually divided into:

Split	Purpose
Training set	Used to fit the model
Validation set	Used to tune hyperparameters
Test set	Used once at the end for final evaluation

The test set should not influence training decisions. If it does, the evaluation becomes overly optimistic.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Feature Scaling

Feature scaling changes feature ranges so one feature does not dominate another.

Common methods:

Method	Formula	Meaning
Standardization	z = (x − μ) / σ	Mean 0 and standard deviation 1
Min-max scaling	x' = (x − xmin) / (xmax − xmin)	Range between 0 and 1

Scaling is important for:

• KNN.
• SVM/SVR.
• Ridge and Lasso.
• Kernel Ridge Regression.
• Neural networks.
• Gradient descent.

Tree-based models usually need less scaling.

Why scale features? Stretched bowls slow gradient descent

1. Formula and symbols

$$L(w)=\tfrac12\sum_j\sigma_j^2(w_j-w_j^\star)^2,\qquad\text{curvatures }\lambda_j=\sigma_j^2$$

$$w\leftarrow w-\eta\nabla L,\quad\eta=\frac{1.6}{\lambda_{\max}},\qquad\kappa=\frac{\lambda_{\max}}{\lambda_{\min}}$$

$$z_j=\frac{x_j-\mu_j}{\sigma_j}\ \Rightarrow\ \lambda_j=1,\ \kappa=1$$

• $x_1$: SNR feature in dB, spread σ₁ = 5
• $x_2$: Bandwidth feature in hundreds of MHz, spread σ₂ = 1
• $w_j$: Weight of feature j; w* is the best weight
• $\sigma_j$: Standard deviation of feature j
• $\lambda_j$: Curvature of the loss along w_j
• $\eta$: Learning rate; it must stay below 2/λ_max or the steep direction diverges
• $\kappa$: Condition number: how stretched the loss bowl is
• $z_j$: Standardised feature (mean 0, spread 1)

The steep direction limits the step size, so the shallow direction crawls. Standardising makes every direction equally steep.

Features

Raw (σ₁ = 5, σ₂ = 1)Standardised

Gradient steps shown

15

2. Work with the current inputs

$$\lambda=(25,\ 1),\quad\kappa=25,\quad\eta=\frac{1.6}{25}=0.064$$

$$\text{per-step factors: }1-\eta\lambda_j=(-0.600,\ 0.936)$$

$$L_{15}=0.2750,\quad\text{steps to }1\%\text{ of }L_0:\ 30$$

3. Result

Condition number κ25

Steps to reach 1% of the start loss30

Loss after 15 steps0.275

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Loss Function

The loss function measures how wrong the model is during training.

For regression, a common loss is mean squared error:

$$\mathrm{MSE}=\frac1n\sum_{i=1}^n(y_i-\widehat y_i)^2$$

• $n$: number of training examples
• $y_{i}$: true value
• $\widehat y_i$: predicted value.
• MSE: average squared prediction error; a large miss receives extra weight.

For classification, a common loss is cross-entropy:

$$L=-\big[y\log p+(1-y)\log(1-p)\big]$$

• $L$: loss for one binary example
• $y$: true class (0 or 1)
• $p$: predicted probability of class 1.
• $\log$: natural logarithm; probabilities should stay strictly between 0 and 1 in this expression.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Effect of Loss Function

Loss choice	Effect
MSE	Strongly punishes large regression errors
MAE	More robust to outliers
Cross-entropy	Standard for classification probabilities
Huber loss	Mixes MSE and MAE behavior
RL reward objective	Optimizes long-term return, not ordinary prediction error

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Learning Rate

The learning rate controls how large each update step is during optimization.

$$\theta_{t+1}=\theta_t-\eta\nabla L(\theta_t)$$

• $\theta _{t}$: parameter vector before update step t.
• $\theta _{t+1}$: parameter vector after update step t.
• $\eta$: learning rate
• $\nabla L(\theta _{t})$: gradient of the loss at the current parameters.

For reference:

Symbol	Meaning
θ	model parameters
η	learning rate
∇L(θ)	gradient of the loss

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Effect of Learning Rate

Learning rate	Effect
Too small	Training is stable but slow
Good value	Training improves smoothly
Too large	Loss may jump, oscillate, or explode

In deep learning, learning-rate schedules often reduce η during training.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Batch Size

Batch size is the number of training examples used before one parameter update.

Batch size	Effect
Small batch	Noisier updates, can generalize well, slower per epoch
Medium batch	Common practical balance
Large batch	Stable gradients, faster hardware use, may generalize worse if not tuned

Example:

$$N=10{,}000,\quad B=100,\quad \lceil N/B\rceil=100.$$

• $N$: training-set size.
• $B$: batch size.

Keep the last partial batch when $N$ is not divisible by $B$.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Epoch

One epoch means the model has seen the full training dataset once.

1 epoch = one complete pass through the training set

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Effect of Epochs

Number of epochs	Effect
Too few	Underfitting
Enough	Good learning
Too many	Possible overfitting

Early stopping can stop training when validation performance stops improving.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Iteration

An iteration is one parameter update.

For mini-batch training:

$$\text{updates per epoch}=\left\lceil\frac NB\right\rceil.$$

This assumes one update per batch and keeps the last partial batch; dropping that batch uses the floor instead.

Epoch and iteration are related, but they are not the same.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Weight

A weight controls how strongly an input feature affects the output.

For a linear model:

$$\widehat y=\beta_0+\sum_{j=1}^{p}\beta_jx_j$$

• $\widehat y$: prediction
• $\beta _{0}$: intercept
• $\beta _{j}$: learned weight for feature xⱼ.
• $p$: number of input features
• $\cdots$ continues the weighted sum.

The coefficients β₁, β₂, ..., βₚ are weights.

Large weights can make a model sensitive. Regularization keeps weights controlled.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Bias

The word bias has two common meanings in machine learning.

References for this section3

• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Bias as an Intercept

In a model equation, bias is the constant term:

$$\widehat y=\beta_0+\beta_1x$$

• $\widehat y$: predicted value
• $x$: input feature
• $\beta _{1}$: its learned weight.
• $\beta _{0}$: intercept, often called bias in this model equation.

It shifts the prediction up or down.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Bias as Model Assumption

Bias can also mean the error caused by overly simple assumptions.

High bias often means:

• The model is too simple.
• Training error is high.
• Validation error is high.
• The model underfits.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Variance

Variance measures how much the model changes when the training data changes.

High variance often means:

• The model is too flexible.
• Training error is low.
• Validation/test error is high.
• The model overfits.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Underfitting

Underfitting happens when the model is too simple to learn the real pattern.

Signs:

• High training error.
• High validation error.
• Model predictions are too crude.

Common fixes:

• Use a more expressive model.
• Train longer.
• Add useful features.
• Reduce excessive regularization.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Overfitting

Overfitting happens when the model learns the training data too closely, including noise.

Signs:

• Very low training error.
• Much higher validation/test error.
• Model performs poorly on new data.

Common fixes:

• Use more data.
• Use regularization.
• Use dropout.
• Use early stopping.
• Use data augmentation.
• Reduce model complexity.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Bias-Variance Tradeoff

The model should be complex enough to learn real structure, but not so complex that it memorizes noise.

Training error	Validation error	Likely interpretation
High	High	Underfitting
Low	High	Overfitting
Low	Low	Good fit

Try the lab: vary the polynomial degree and compare training with validation MSE. Low training error alone does not show a useful model. This single fixed split illustrates model flexibility; statistical bias and variance require repeated datasets.

Does a more flexible model always validate better?

1. Formula and symbols

$$\widehat y(x)=\sum_{j=0}^{p}\beta_jx^j$$

$$(X^TX+\lambda P)\beta=X^Ty,\quad P=\operatorname{diag}(0,1,\ldots,1)$$

$$\mathrm{MSE}=\frac1N\sum_i(\widehat y_i-y_i)^2$$

• $p$: degree
• $X$: polynomial design matrix
• $\beta$: learned coefficients
• $y$: training targets
• $\lambda$: penalty
• $P$: excludes the intercept from that penalty
• $N$: Number of examples in the set being scored

MSE is computed separately on training and validation sets of size N.

Polynomial degree

3

Target noise standard deviation

0.2

2. Work with the current inputs

$$(X^TX+1.00\times10^{-6}P)\beta=X^Ty,\quad p=3$$

$$\mathrm{MSE}_{\rm train}=\frac1{12}\sum_i(\widehat y_i-y_i)^2=0.0256$$

$$\mathrm{MSE}_{\rm val}=\frac1{40}\sum_j(\widehat y_j-y_j)^2=0.0672$$

3. Result

Training MSE0.0256

Validation MSE0.0672

Prediction at x = 0.51.003

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Regularization

Regularization discourages unnecessary complexity.

Common types:

Method	Effect
L1 regularization	Can push some weights to zero
L2 regularization	Shrinks weights smoothly
Dropout	Randomly disables neurons during training
Early stopping	Stops before validation performance gets worse
Data augmentation	Creates more varied training examples

Ridge Regression uses L2 regularization:

$$\min_{\beta_0,\beta}\left[\sum_i(y_i-\widehat y_i)^2+\lambda\sum_{j=1}^p\beta_j^2\right]$$

• $i$: example index
• $y_{i} - \widehat y_{i}$: prediction error
• $j$: weight index
• $\beta _{j}$: model weight.
• $\lambda$: nonnegative regularization strength; the first sum fits data, the second penalizes large weights.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Dropout

Dropout is mainly used in neural networks. During training, some neurons are randomly ignored.

Effect:

• Reduces dependence on individual neurons.
• Helps prevent overfitting.
• Can slow training slightly.

Dropout is usually turned off during final evaluation.

Dropout: train with a random half-network, test with the whole

1. Formula and symbols

$$m_j\sim\mathrm{Bernoulli}(p),\qquad\tilde h_j=\frac{m_j\,h_j}{p}\quad\text{(training)}$$

$$\mathbb E[\tilde h_j]=h_j\ \Rightarrow\ \mathbb E\Big[\sum_jv_j\tilde h_j\Big]=\sum_jv_jh_j$$

• $h_j$: Activation of hidden neuron j
• $m_j$: Random mask: 1 keeps the neuron for this step, 0 drops it
• $p$: Keep probability (drop rate = 1 − p)
• $\tilde h_j$: Activation actually passed on, scaled by 1/p (inverted dropout)
• $v_j$: Output weight of neuron j

At inference time dropout is switched off and no scaling is needed, because the 1/p factor during training already kept the average output unchanged.

Keep probability p

0.6

Phase

Training (dropout on)Inference (dropout off)

Training step (pauses)

1

2. Work with the current inputs

$$m=(0,0,1,0,0,1,1,0),\quad\text{scale}=1/0.6=1.667$$

$$y=\sum_jv_jm_jh_j\cdot1.667=1.700,\quad\overline{y}_{2000\ \rm masks}=1.781\approx y_{\rm full}=1.850$$

3. Result

Active neurons this step3 of 8

Output this step1.700full network: 1.850

Average over 2000 masks1.781inverted dropout keeps the mean

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

Optimizer

An optimizer decides how parameters are updated.

Optimizer	Common use
SGD	Simple, strong baseline
Momentum	Smooths noisy updates
RMSProp	Useful for non-stationary gradients
Adam	Common default in deep learning
AdamW	Adam with better weight decay behavior

Optimizer choice affects training speed and stability.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Activation Function

Activation functions introduce non-linearity in neural networks.

Without nonlinear activations, a stack of affine layers is equivalent to one affine model.

$$a=g\!\left(\beta_0+\sum_{j=1}^{p}\beta_jx_j\right)$$

• $a$: neuron output
• $g$: activation function applied after the weighted sum.
• $x_{j}$: input feature j.
• $\beta _{j}$: learned weight for feature j.
• $\beta _{0}$: intercept
• $p$: feature count.

Activation	Effect
Sigmoid	Outputs 0 to 1, can saturate
Tanh	Outputs -1 to 1
ReLU	Fast and common, can create dead neurons
GELU	Common in transformer models
Softmax	Converts class scores into probabilities

Activation functions and their slopes

1. Formula and symbols

$$\sigma(z)=\frac1{1+e^{-z}}$$

$$\sigma'(z)=\sigma(z)(1-\sigma(z))\le0.25$$

$$\text{gradient through 10 such layers}\ \propto\ |a'(z)|^{10}$$

• $z$: Neuron input: the weighted sum before activation
• $a(z)$: Activation output passed to the next layer
• $a'(z)$: Slope: how much a small change in z changes the output (used by backpropagation)

Without a non-linear activation, any stack of layers collapses into one linear map. The slope matters as much as the shape: where it is near zero, learning stalls.

Activation

SigmoidTanhReLULeaky ReLUGELU

Neuron input z

2

2. Work with the current inputs

$$a(2.0)=0.8808,\qquad a'(2.0)=0.1050$$

$$|a'(z)|^{10}=1.63\times10^{-10}$$

3. Result

Output a(z)0.881

Slope a′(z)0.105gradient passes

Slope over 10 layers1.6e-10

References for this section3

• Scikit-learn documentation, "Model evaluation".
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

DNN-Specific Parameters

Parameter	Effect
Number of layers	More depth can learn more complex patterns but can overfit
Hidden units	More units increase capacity
Batch normalization	Stabilizes layer inputs
Layer normalization	Common in transformers
Residual connection	Helps train deep networks
Weight initialization	Affects early training stability
Gradient clipping	Prevents exploding gradients

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• Scikit-learn documentation, "Model evaluation".
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.

ML vs. DL vs. DNN vs. DRL

Term	Meaning
ML	General field where models learn from data
DL	Deep learning using neural networks with many layers
DNN	Deep neural network architecture
RL	Reinforcement learning through actions and rewards
DRL	Deep reinforcement learning, where deep networks are used inside RL

In short:

DL is a part of ML.
DNN is a type of DL model.
DRL combines deep learning with reinforcement learning.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• Scikit-learn documentation, "Model evaluation".

DRL-Specific Parameters

Deep Reinforcement Learning has extra parameters because the agent learns by interacting with an environment.

Term	Meaning
State s	What the agent observes
Action a	What the agent chooses
Reward r	Feedback from the environment
Policy π(a given s)	Rule for choosing actions
Episode	One full interaction run
Return	Total future reward

The common discounted return is:

$$G_t=r_t+\gamma r_{t+1}+\gamma^2r_{t+2}+\cdots$$

• $G_{t}$: discounted return starting at step t
• $r_{t}$: reward at that step.
• $\gamma$: discount factor, usually between 0 and 1; later rewards receive powers of γ.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Important DRL Hyperparameters

Hyperparameter	Effect
Discount factor γ	Controls how much future rewards matter
Exploration rate ε	Controls random exploration in epsilon-greedy methods
Replay buffer size	Controls how many past experiences are stored
Target-network update rate	Stabilizes Q-learning-style methods
Reward scaling	Changes the magnitude of learning signals
Episode length	Controls how far the agent can interact before reset

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Effects in DRL

Setting	If too small	If too large
γ	Agent becomes short-sighted	Agent may overvalue distant uncertain rewards
ε	Agent may stop exploring too early	Agent may behave too randomly
Replay buffer	Learns from too little experience	Uses more memory and may learn from stale data
Learning rate	Slow learning	Unstable Q-values or policy updates
Reward scale	Weak gradients	Exploding or unstable updates

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Practical Tuning Order

For many ML/DL projects, tune in this order:

1. Fix the data split and random seed.
2. Choose a simple baseline.
3. Check underfitting or overfitting.
4. Tune learning rate.
5. Tune batch size and epochs.
6. Add regularization if overfitting.
7. Increase model capacity only when the model underfits.
8. Test with multiple seeds.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Quick Diagnosis Table

Observation	Likely issue	What to try
Training and validation errors are both high	Underfitting	Bigger model, better features, lower regularization
Training error is low but validation error is high	Overfitting	More data, regularization, dropout, early stopping
Loss is unstable or explodes	Learning rate too high	Lower learning rate, gradient clipping
Training is very slow	Learning rate too low or model too large	Increase learning rate carefully, simplify model
Results change a lot across runs	Seed sensitivity	Run multiple seeds and report mean/std
DRL reward is noisy	Exploration or environment variance	Tune ε, replay buffer, reward scaling

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Takeaway

• Training setup: seed and train/validation/test split make comparisons reproducible.
• Optimization: learning rate, batch size, and epochs control how the model updates.
• Generalization: bias, variance, overfitting, underfitting, and regularization explain unseen-data performance.
• Deep RL: also track exploration, discount factor, replay buffer, and reward design.

Continue with Gradient Descent to build on this note.

References for this section3

• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.