Skip to content
New
Latest News || 27 October 2026: Dr Soumava Mukherjee's invited talk at Queen's University Belfast joined device-level hardware innovation with channel-level control. || 22 October 2026: I received a Certificate of Appreciation for serving as a reviewer for IEEE IMAS 2026. || 15 October 2026: I presented a poster on beam-domain ML fingerprinting for RIS-aided mmWave localisation at the Centre for Wireless Innovation's annual symposium. || 15 October 2026: CWI's annual symposium brought research and industry perspectives on 6G, ISAC, optical wireless, AI, and future networks to Queen's University Belfast. || 14 October 2026: Erik Nordell of Ericsson visited our Keysight Lab for a live RIS-assisted video streaming demonstration.
Tarek Hassan
Knowledge Hubmachine learning11. Transfer Learning

machine learning learning note

11. Transfer Learning

A practical guide to transfer learning, frozen layers, fine-tuning, applications, benefits, and limitations.

At a glance

  • Reuse a source representationA pre-trained backbone may reduce target-data and training needs; negative transfer is possible when domains differ.
  • Freezing changes training stateFrozen coefficients stay fixed. A new head remains trainable; frozen layers still participate in the forward pass.
  • Validate how much to unfreezeTrain the head first, optionally fine-tune with a low learning rate, and recompile Keras after changing trainability.

What is Transfer Learning?

Transfer learning starts from a model trained on one task and adapts it to a related task.

  • Source task: where the pre-trained representation came from.

  • Target task: the new problem we care about.

  • Why use it: fewer target labels or less training time may be needed.

  • Example: a broad image model may already detect edges and textures.

  • Reuse: adapt those features for a smaller medical, defect, or remote-sensing dataset.

  • Check: improvement is not guaranteed when source and target images differ greatly.

large source task -> pre-trained model -> adapted target task
References for this section3

Why Transfer Learning Matters

Training a deep model from zero can be expensive because it usually needs a large dataset, strong hardware, and long training time. Transfer learning reduces that burden by reusing a model that already has general-purpose representations.

Important advantages:

  • Less data required: The target task can benefit from features learned on a larger source task.
  • Faster training: Fewer parameters may need to be trained from scratch.
  • Better starting point: Pre-trained weights often produce better early performance than random initialization.
  • Improved generalization: General features learned from broad datasets can reduce overfitting on small target datasets.
  • Lower cost: Reusing existing models can save computation, time, and development effort.
References for this section3

How Transfer Learning Works

Most transfer-learning workflows follow four steps:

  1. Choose a pre-trained model. Start with a model trained on a large source dataset, such as ImageNet for vision or a large text corpus for natural language processing.
  2. Reuse the base model. Keep the early and middle layers that already learned reusable representations.
  3. Replace the task-specific head. Remove or replace the final classifier or prediction layer so the model matches the new target labels.
  4. Train or fine-tune. Train the new head first, then optionally unfreeze some deeper layers to adapt the model more closely to the target data.

For image models, early layers often learn generic features such as edges and textures. Deeper layers learn more task-specific patterns such as object parts or class-level concepts.

References for this section3

Frozen and Trainable Layers

A central design choice in transfer learning is deciding which layers should stay frozen and which layers should be updated.

AspectFrozen LayersTrainable Layers
MeaningWeights are kept fixed during training.Weights are updated during training.
PurposePreserve general knowledge from the source task.Adapt the model to the target task.
ComputationLower training cost because fewer parameters change.Higher cost because more parameters are optimized.
Typical useSmall or similar target datasets.Larger or different target datasets.
CNN exampleEarly convolution layers that detect edges and textures.Later convolution or dense layers that learn task-specific patterns.

How much training state changes when layers are frozen?

1. Formula and symbols
Nℓ=(din+1)doutN_\ell=(d_{\rm in}+1)d_{\rm out}
Ntrainable=∑ℓ unfrozenNℓN_{\rm trainable}=\sum_{\ell\ \rm unfrozen}N_\ell
M=4(Ntotal+2Ntrainable) bytesM=4(N_{\rm total}+2N_{\rm trainable})\ \mathrm{bytes}
  • dind_{\rm in}: Number of inputs to the dense layer
  • doutd_{\rm out}: Number of outputs from the dense layer
  • NℓN_\ell: coefficients including biases
  • NtotalN_{\rm total}: all coefficients
  • NtrainableN_{\rm trainable}: Coefficients in the unfrozen layers
  • MM: Weight and optimizer-state memory in bytes

The memory model counts float32 weights and two Adam moment states per trainable coefficient, excluding gradients and activations.

32 → 642112 coefficientsFrozen64 → 1288320 coefficientsFrozen128 → 2258 coefficientsTrainableSource backbone → new target head; freezing is a training choiceTarget validation results must determine whether transfer helps.
Frozen backbone blocks
2. Work with the current inputs
N1=(32+1)64=2112,N2=(64+1)128=8320N_1=(32+1)64=2112,\quad N_2=(64+1)128=8320
Nhead=(128+1)2=258,Ntrainable=258N_{\rm head}=(128+1)2=258,\quad N_{\rm trainable}=258
M=4(Ntotal+2Ntrainable)1024=43.8 KiBM=\frac{4(N_{\rm total}+2N_{\rm trainable})}{1024}=43.8\ \mathrm{KiB}
3. Result
Trainable / total parameters258 / 10,690
Trainable fraction2.4%
Weights + two moment states43.8 KiB
Parameter accounting for a pre-trained dense backbone 32→64→128 and a new 128→C classifier. Each dense layer includes biases. Freeze zero, one or both backbone blocks; the head is always trainable. Memory counts float32 weights plus two Adam moments for trainable coefficients only, excluding gradients, activations and implementation overhead. Fewer trainable parameters do not guarantee faster inference or better target accuracy.
References for this section3

Choosing What to Freeze

The decision depends mainly on dataset size and similarity between the source and target domains.

References for this section3

Small and Similar Dataset

Freeze most of the base model and train only the new output head. This limits overfitting and keeps the general features intact.

Example: adapting an ImageNet model to classify a small set of everyday object categories.

References for this section3

Large and Similar Dataset

Unfreeze more layers and fine-tune part of the model. Because there is enough data, the model can adapt without immediately overfitting.

Example: adapting a general image classifier to a large industrial image dataset.

References for this section3

Small and Different Dataset

Practical sequence:

  1. Freeze early layers and train a new task-specific head.
  2. If validation improves, fine-tune selected later layers with a low learning rate.
  3. Compare against a simple from-scratch baseline to detect negative transfer.

Example: adapting a natural-image model to a small medical-imaging dataset.

References for this section3

Large and Different Dataset

Fine-tune more of the model, or even the whole model, because the target data is large enough to support deeper adaptation.

Example: adapting a general visual backbone to satellite imagery or specialized scientific images.

References for this section3

Common Workflow

A simple deep-learning workflow looks like this:

load pre-trained backbone
remove original classifier
add new classifier for target labels
freeze backbone
train new classifier
unfreeze selected layers
fine-tune with a small learning rate
evaluate on target test data

In Keras, this model expects RGB pixel values in [0,255] and one-hot target labels. It downloads ImageNet weights; supply your own training and validation datasets for the commented fitting call. Keeping the backbone call at training=False preserves BatchNorm inference behavior during fine-tuning.

import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Input, Rescaling
from tensorflow.keras.models import Model

num_classes = 3  # Replace with the target class count.

base_model = MobileNetV2(
    weights="imagenet",
    include_top=False,
    input_shape=(224, 224, 3),
)
base_model.trainable = False

inputs = Input(shape=(224, 224, 3))
x = Rescaling(1.0 / 127.5, offset=-1)(inputs)
x = base_model(x, training=False)
x = GlobalAveragePooling2D()(x)
outputs = Dense(num_classes, activation="softmax")(x)

model = Model(inputs, outputs)
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
    loss="categorical_crossentropy",
    metrics=["accuracy"],
)
# With your own (image, one-hot label) datasets:
# model.fit(train_ds, validation_data=validation_ds, epochs=5)

After the new classifier learns a useful mapping, selected layers can be unfrozen:

base_model.trainable = True

for layer in base_model.layers[:-20]:
    layer.trainable = False

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-5),
    loss="categorical_crossentropy",
    metrics=["accuracy"],
)

Recompile after changing layer trainability. Then fit again using the same target training/validation split. Freezing does not remove forward-pass work.

The smaller learning rate matters because fine-tuning should adjust the pre-trained features gently rather than overwrite them.

References for this section3

Applications

Transfer learning is widely used across modern AI:

  • Computer vision: Image classification, object detection, facial recognition, medical-image analysis, remote sensing, and quality inspection.
  • Natural language processing: Text classification, sentiment analysis, question answering, summarization, and information extraction using pre-trained language models.
  • Healthcare: Diagnostic support from X-rays, MRI, CT, pathology slides, and other data-scarce medical datasets.
  • Finance: Fraud detection, credit scoring, risk prediction, and anomaly detection where patterns learned from one dataset can support related tasks.
  • Wireless communications: Beam selection, localization, channel estimation, and resource allocation when a model trained in one environment is adapted to another deployment scenario.
References for this section3

Advantages

  • Training is faster because the model starts with useful representations.
  • Performance can improve when the target dataset is small.
  • Data requirements are lower than full training from scratch.
  • Compute cost can drop because only part of the model may need training.
  • Deployment is easier when well-tested pre-trained backbones are available.
References for this section3

Limitations

Transfer learning is powerful, but it is not automatic magic.

  • Domain mismatch: If source and target data are too different, transferred features may be weak or misleading.
  • Negative transfer: Reusing unsuitable knowledge can reduce performance.
  • Overfitting: Fine-tuning too many parameters on a small dataset can harm generalization.
  • Hardware demand: Large pre-trained models may still require significant memory and compute.
  • Bias transfer: A model can carry biases from the source dataset into the target application.
References for this section3

Practical Tips

  • Start with a frozen backbone and train the new classifier first.
  • Use a smaller learning rate during fine-tuning.
  • Unfreeze gradually instead of training every layer immediately.
  • Use validation performance to decide how many layers to unfreeze.
  • Keep data preprocessing consistent with the pre-trained model.
  • Compare against a simple baseline so transfer learning is actually earning its keep.
References for this section3

Summary

  • Main decision: how much of the pre-trained model to freeze or fine-tune.
  • Evidence: target validation/test results must show that transfer helps under the same data and compute budget.
References for this section3

Check your understanding

0 of 4 answered

  1. A dense layer maps 32 inputs to 64 outputs with biases. How many coefficients does it have?

  2. A new 128→3 dense head includes biases. How many trainable coefficients does the head have?

  3. What is necessary after changing Keras layer trainability before continuing training?

  4. Does freezing the backbone alone reduce its forward-pass coefficient count?

Complete references and further reading