9. Deep Learning Architectures

What is Deep Learning?

Deep learning uses neural networks with multiple learned layers.

• Early layers: can detect simpler patterns.
• Later layers: combine them into task-relevant representations.
• Physical caution: what a layer learns depends on data and training; it is not guaranteed to match a human concept.

For images, early layers may detect edges. Middle layers may detect shapes and textures. Deeper layers may detect objects. For language, early layers may learn local word patterns, while deeper layers learn meaning and context.

LeCun, Bengio, and Hinton described deep learning as using multiple processing layers to learn data representations with multiple levels of abstraction.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Why Deep Learning Became Powerful

Deep learning became dominant because three things arrived together:

• Large datasets: Images, text, speech, sensor data, and web-scale corpora.
• Computation: GPUs and distributed training made large neural networks practical.
• Algorithmic improvements: Better activation functions, initialization, normalization, optimization, and architectures.

Traditional ML often requires manual feature engineering. Deep learning can learn features directly from raw or lightly processed data.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

The Neuron

A neuron receives inputs, multiplies them by weights, adds a bias, and passes the result through an activation function.

$$a=g\!\left(\beta_0+\sum_{j=1}^{p}\beta_jx_j\right)$$

• $x_{j}$: input feature j.
• $\beta _{j}$: learned weight for feature j.
• $\beta _{0}$ is the bias.
• $g$ converts the weighted sum to output a
• $p$ is the number of inputs.

The activation function introduces non-linearity. Without non-linearity, stacking layers would still behave like one linear model.

Common activation functions:

• Sigmoid: Maps values to 0-1, but can suffer from vanishing gradients.
• Tanh: Maps values to -1 to 1.
• ReLU: Simple and widely used; outputs zero for negative inputs.
• GELU: Common in transformer models.

Where does a neural-network output and its gradient come from?

1. Formula and symbols

$$z_1=x+y-0.2,\quad z_2=x-y+0.1$$

$$p=\sigma(g[a(z_1)-a(z_2)])$$

$$\frac{\partial p}{\partial x}=p(1-p)g[a'(z_1)-a'(z_2)]$$

• $x$: First input feature
• $y$: Second input feature
• $z$: hidden weighted sums
• $a$: selected activation
• $g$: output gain
• $\sigma$: sigmoid
• $p$: output value

Constants are the fixed demonstration weights/biases

the derivative is local sensitivity.

Input x

0.1

Input y

0.6

Output gain g

2

Hidden activation a

ReLUTanhSigmoid

2. Work with the current inputs

$$z_1=x+y-0.2=0.500,\quad z_2=x-y+0.1=-0.400$$

$$p=\sigma(g[a(z_1)-a(z_2)])=\sigma(1.000)=0.731$$

$$\frac{\partial p}{\partial x}=p(1-p)g[a'(z_1)-a'(z_2)]=0.39322$$

3. Result

Output sigmoid value0.731

Hidden outputs h₁ / h₂0.500 / 0.000

Local derivative ∂p/∂x0.39322

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

1. Artificial Neural Networks

An Artificial Neural Network (ANN), also called a multilayer perceptron when fully connected, is the basic neural network architecture.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Best For

• Tabular data.
• Simple non-linear prediction.
• Baseline deep learning experiments.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

How It Works

Each neuron in one layer connects to neurons in the next layer. Information flows forward from input to output. During training, backpropagation computes gradients, and an optimizer updates the weights.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Strengths

• Learns non-linear functions.
• Flexible and general.
• Easy to build compared with specialized networks.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Weaknesses

• Does not naturally exploit image structure or sequence order.
• Can overfit small datasets.
• Requires scaling and careful training.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

2. Deep Neural Networks

A Deep Neural Network (DNN) is an ANN with many hidden layers. Depth allows the model to learn hierarchical representations.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Intuition

One layer learns simple combinations. Several layers can build more complex functions by composing simple ones.

Example for image recognition:

pixels -> edges -> corners -> parts -> object

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Training Challenges

Deep networks can be hard to train because gradients may vanish, explode, or become noisy. Modern networks use tools such as:

• Careful initialization.
• ReLU or GELU activations.
• Batch normalization or layer normalization.
• Dropout.
• Residual connections.
• Adam or AdamW optimizers.

Why do deep networks need careful activations and initialisation?

1. Formula and symbols

$$h_\ell=a(z_\ell),\qquad z_\ell=W_\ell h_{\ell-1}$$

$$\delta_\ell=a'(z_\ell)\odot\big(W_{\ell+1}^\top\delta_{\ell+1}\big),\qquad\lVert\delta_1\rVert\approx\lVert\delta_L\rVert\prod_{\ell}\big(s\,\overline{|a'|}\big)$$

• $h_\ell$: Output of layer ℓ (8 neurons wide)
• $W_\ell$: Weight matrix of layer ℓ, drawn with standard deviation s/√8
• $a,\ a'$: Activation function and its derivative
• $\delta_\ell$: Gradient signal that reaches layer ℓ during backpropagation
• $L$: Number of layers
• $s$: Weight-initialisation scale

Each layer multiplies the backward signal by roughly s·|a′|. Sigmoid has |a′| ≤ 0.25, so deep sigmoid stacks shrink the signal geometrically; a large s can make it grow instead.

Hidden layers L

8

Activation

SigmoidTanhReLU

Initialisation scale s

1

2. Work with the current inputs

$$\lVert\delta_1\rVert=8.33\times10^{-7},\quad\lVert\delta_{8}\rVert=2.43\times10^{-1}$$

$$\frac{\lVert\delta_1\rVert}{\lVert\delta_L\rVert}=3.43\times10^{-6}\quad(vanishing)$$

3. Result

Gradient at first layer8.3e-7

First / last layer ratio3.4e-6vanishing

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

3. Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are designed for grid-like data, especially images.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Why CNNs Work Well for Images

Images have local structure. Nearby pixels are related. A CNN uses small filters that slide across the image and detect local patterns such as edges, textures, and corners.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Main Components

• Convolution: Applies filters to detect patterns.
• Activation: Adds non-linearity.
• Pooling: Reduces spatial size and improves robustness.
• Fully connected layer: Uses learned features for final prediction.

A CNN filter sliding over an image

1. Formula and symbols

$$Y[i,j]=\sum_{a=0}^{2}\sum_{b=0}^{2}K[a,b]\;X[iS-P+a,\ jS-P+b]$$

$$N_{\rm out}=\Big\lfloor\frac{N-k+2P}{S}\Big\rfloor+1,\qquad\text{pool: }Z[i,j]=\max_{2\times2}\mathrm{ReLU}(Y)$$

• $X$: Input image, N × N = 8 × 8 pixels (1 = bright, 0 = dark)
• $K$: 3 × 3 filter (kernel): 9 shared weights
• $Y$: Feature map: how strongly the pattern appears at each position
• $k$: Kernel size (3)
• $S$: Stride: how many pixels the window jumps
• $P$: Zero padding added around the border
• $Z$: Pooled map: the largest positive response in each 2 × 2 block

Deep-learning libraries call this operation convolution, although mathematically it is cross-correlation (the kernel is not flipped).

Filter

Vertical edgeHorizontal edgeBlurSharpen

Stride S

12

Padding P

01

Window position (pauses)

row 1, column 1

2. Work with the current inputs

$$N_{\rm out}=\Big\lfloor\frac{8-3+2(0)}{1}\Big\rfloor+1=6,\qquad\text{pooled: }3\times3$$

$$Y[1,1]=0=0.00$$

$$\text{parameters: }3\times3+1=10\ \text{(shared)}\quad\text{vs dense }64\times36=2304$$

3. Result

Feature-map size6 × 6after pooling: 3 × 3

Response at the window0.00dark→bright top to bottom

Weights in this layer10a dense layer would need 2304

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Best For

• Image classification.
• Object detection.
• Medical imaging.
• Remote sensing.
• Spectrogram and signal-pattern recognition.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

4. Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are designed for sequential data. They process one step at a time and maintain a hidden state that carries information from previous steps.

An RNN unrolled in time: how long does it remember?

1. Formula and symbols

$$h_t=\tanh(w\,h_{t-1}+u\,x_t),\qquad h_0=0$$

$$\frac{\partial h_T}{\partial h_t}=\prod_{k=t+1}^{T}w\,(1-h_k^2)$$

• $x_t$: Input at step t (one event at t = 1, then zeros)
• $h_t$: Hidden state: the network’s memory after step t
• $w$: Recurrent weight: how much of the old memory is kept
• $u$: Input weight
• $T$: Sequence length
• $\partial h_T/\partial h_t$: How much the last state still depends on step t; training signals travel back through this product

The same w and u are reused at every step. If |w(1 − h²)| < 1 the product shrinks geometrically: the plain RNN forgets, which LSTM and GRU gates were designed to fix.

Recurrent weight w

0.8

Input weight u

1.5

Sequence length T

9

Highlighted step (pauses)

1

2. Work with the current inputs

$$h_{1}=\tanh(0.8\times0.000+1.5\times1)=0.905$$

$$\frac{\partial h_{9}}{\partial h_1}=\prod_{k=2}^{9}0.8(1-h_k^2)=5.94\times10^{-2}$$

3. Result

Memory left at step 90.108started at h₁ = 0.905

Dependence of h_T on h₁5.9e-2

Steps with dependence ≥ 0.18 of 9

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Best For

• Time series.
• Speech.
• Sensor streams.
• Sequential control signals.

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Limitations

Standard RNNs struggle with long-range dependencies. LSTM and GRU architectures were designed to remember information for longer periods, but transformers now dominate many sequence modeling tasks.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

5. Transformers

Transformers are neural networks built around attention mechanisms. Instead of processing a sequence strictly step by step, attention allows the model to compare each token with other tokens directly.

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Why Attention Matters

In a sentence, the meaning of one word may depend on another word far away. Attention helps the model decide which parts of the input are most relevant.

Self-attention: which words does each word look at?

1. Formula and symbols

$$s_{ij}=\frac{q_i^\top k_j}{\sqrt{d_k}},\qquad\alpha_{ij}=\frac{e^{s_{ij}}}{\sum_{j'}e^{s_{ij'}}}$$

$$o_i=\sum_j\alpha_{ij}v_j\qquad\text{(causal mask: }\alpha_{ij}=0\text{ for }j>i)$$

• $q_i$: Query of token i: what this token is looking for
• $k_j$: Key of token j: what token j offers (here k_j = c·e_j)
• $v_j$: Value of token j: the content passed on (here v_j = e_j)
• $d_k$: Key dimension (4)
• $s_{ij}$: Scaled similarity score
• $\alpha_{ij}$: Attention weight: share of token i’s focus on token j (each row sums to 1)
• $o_i$: New representation of token i: a weighted mix of the values

The vectors are hand-set along four readable directions (device, person, action, grammar) to show the mechanism; a trained transformer learns them, uses hundreds of dimensions and many heads in parallel.

Query word

1. The2. beam3. missed4. the5. user6. because7. it8. moved

Key scale c (score sharpness)

3

Causal mask

Off (encoder)On (decoder)

2. Work with the current inputs

$$s_{7,j}=\frac{q_{7}^\top(3e_j)}{\sqrt4},\quad\alpha_{7,\cdot}=(0.04,\ 0.05,\ 0.05,\ 0.04,\ 0.40,\ 0.04,\ 0.27,\ 0.12)$$

$$o_{7}=\sum_j\alpha_{7j}e_j=(0.15,\ 0.68,\ 0.18,\ 0.22)\ \ [\text{device, person, action, grammar}]$$

3. Result

“it” attends most to“user” (40%)then “it” 27%

Strongest meaning in outputperson

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Best For

• Language models.
• Machine translation.
• Text classification.
• Vision transformers.
• Multimodal models combining text, images, and audio.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Training a Deep Network

The training loop is usually:

1. Send a batch of data through the network.
2. Compute the loss.
3. Use backpropagation to compute gradients.
4. Update weights using an optimizer.
5. Repeat over many batches and epochs.

Backpropagation became a key foundation for modern neural networks after the work of Rumelhart, Hinton, and Williams in 1986.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Regularization

Deep networks can memorize training data. Regularization helps them generalize.

Common techniques:

• Dropout: Randomly disables neurons during training.
• Weight decay: Penalizes overly large weights.
• Data augmentation: Creates modified training examples.
• Early stopping: Stops training when validation performance stops improving.
• Batch/layer normalization: Stabilizes training.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

When Deep Learning is a Good Choice

Deep learning is usually strong when:

• The dataset is large.
• The input is complex, such as image, audio, text, or raw signal data.
• Manual feature engineering is difficult.
• High predictive performance is more important than simple interpretability.

It may not be the best first choice when:

• The dataset is small.
• Interpretability is the main requirement.
• The problem is simple tabular prediction.
• Compute resources are limited.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.

Python: Load a DNN with Keras

Use a DNN when your features are tabular or already converted into numeric vectors. The Sequential API is the simplest way to stack layers.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Input(shape=(20,)),
    layers.Dense(64, activation="relu"),
    layers.Dropout(0.2),
    layers.Dense(32, activation="relu"),
    layers.Dense(1, activation="sigmoid"),
])

model.compile(
    optimizer="adam",
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

# X_train shape: (samples, 20), y_train shape: (samples,)
# model.fit(X_train, y_train, validation_split=0.2, epochs=20, batch_size=32)

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Python: Load a CNN for Image Classification

CNNs expect image-like tensors. For RGB images of size 128 x 128, the input shape is (128, 128, 3).

from tensorflow import keras
from tensorflow.keras import layers

cnn = keras.Sequential([
    layers.Input(shape=(128, 128, 3)),
    layers.Rescaling(1.0 / 255),
    layers.Conv2D(32, 3, activation="relu"),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation="relu"),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(128, activation="relu"),
    layers.Dense(10, activation="softmax"),
])

cnn.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Python: Load an RNN/LSTM for Sequence Data

For sequence data, the input shape is usually (time_steps, features_per_step).

from tensorflow import keras
from tensorflow.keras import layers

lstm_model = keras.Sequential([
    layers.Input(shape=(50, 8)),
    layers.LSTM(64),
    layers.Dense(32, activation="relu"),
    layers.Dense(1),
])

lstm_model.compile(
    optimizer="adam",
    loss="mse",
    metrics=["mae"]
)

References for this section3

• Keras documentation, "The Sequential model".
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.

Python: Save and Load a Deep Learning Model

model.save("dnn_model.keras")

loaded_model = keras.models.load_model("dnn_model.keras")
# predictions = loaded_model.predict(X_test)

For real projects, keep preprocessing with the model or document it carefully. A neural network trained on normalized data will perform badly if deployed on raw unnormalized data.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• Keras documentation, "The Sequential model".

Takeaway

• CNN: useful for spatial grids such as images.
• RNN: models ordered sequences.
• Transformer: uses attention to relate positions in a sequence.
• Cost: data, compute, regularization, and careful held-out evaluation.

Continue with Reinforcement Learning and Deep RL to build on this note.

References for this section3

• Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning", Nature, vol. 521, pp. 436-444, 2015.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, pp. 533-536, 1986.