7. Ridge and Kernel Ridge Regression

The Big Picture

Ridge Regression and Kernel Ridge Regression are both used for regression, which means they predict a number.

Examples:

• Predict house price.
• Predict temperature.
• Predict received signal strength.
• Predict localization error in meters.
• Predict channel gain.

The important question is:

Given some input features, what numerical value should the model predict?

Ridge Regression starts from Linear Regression and makes it more stable. Kernel Ridge Regression starts from Ridge Regression and makes it able to learn curved, non-linear patterns.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Start from Linear Regression

Linear Regression assumes the output can be predicted by adding weighted input features.

For one feature:

$$\widehat y=\beta_0+\beta_1x$$

• $\widehat y$ is the prediction
• $x$: input feature.
• $\beta _{1}$: learned slope.
• $\beta _{0}$: intercept.

For many features:

$$\widehat y=\beta_0+\sum_{j=1}^{p}\beta_jx_j$$

• $x_{j}$: input feature j.
• $\beta _{j}$: learned weight for feature j.
• $p$: number of features.
• $\beta _{0}$ is the intercept.

Where:

Symbol	Meaning
ŷ	predicted output
x₁, x₂, ..., xₚ	input features
β₁, β₂, ..., βₚ	learned coefficients or weights
β₀	bias or intercept

For example, in house-price prediction:

$$\begin{aligned}\widehat{\mathrm{price}}={}&\beta_0+\beta_1\mathrm{area}\\&+\beta_2\mathrm{bedrooms}+\beta_3\mathrm{distance}\end{aligned}$$

• $\mathrm{price}$ is the prediction; area, bedrooms, and distance are input features.
• $\beta _{1}$: learned coefficient of the first feature.
• $\beta _{2}$: learned coefficient of the second feature.
• $\beta _{3}$: learned coefficient of the third feature.
• $\beta _{0}$ is the baseline price.

Training means finding the best weights so the predictions are close to the true values.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn, Common pitfalls: data leakage and pipelines.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.

How Linear Regression Learns

The model makes predictions, compares them with the true values, and tries to reduce the error.

For ordinary Linear Regression, the most common objective is:

$$\min_{\beta_0,\beta}\sum_i(y_i-\widehat y_i)^2$$

• $y_{i}$: measured target for sample i.
• $\widehat y_i$: model prediction for sample i.
• $\sum _{i}$ totals squared errors.

In words:

make the squared prediction errors as small as possible

This is called ordinary least squares.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn, Common pitfalls: data leakage and pipelines.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.

The Problem with Ordinary Linear Regression

Linear Regression can become unstable when:

• The dataset is small.
• The dataset contains noise.
• Some features are strongly correlated.
• There are too many features compared with the number of samples.
• The model tries too hard to fit every training point.

When this happens, the model may learn very large weights.

Large weights often mean:

small change in input -> large change in prediction

That can make the model sensitive, unstable, and poor on new data.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn, Common pitfalls: data leakage and pipelines.
• Scikit-learn documentation, "Ridge".

What Ridge Regression Adds

Ridge Regression adds a penalty for large weights.

Ordinary Linear Regression:

$$\min\ \mathrm{prediction\ error}$$

• Prediction error measures disagreement between measured and predicted values; ordinary least squares uses the squared-error sum above.

Ridge Regression:

$$\min\big[\mathrm{prediction\ error}+\mathrm{weight\ penalty}\big]$$

• Prediction error rewards a good fit; weight penalty discourages large coefficients.

A common Ridge objective is:

$$\min_{\beta_0,\beta}\left[\sum_i(y_i-\widehat y_i)^2+\lambda\sum_{j=1}^p\beta_j^2\right]$$

• $y_{i}$: measured target for sample i.
• $\widehat y_i$: model prediction for sample i.
• $\beta _{j}$ is feature weight j.
• $\lambda ≥ 0$ sets how strongly the squared weights are penalized; the intercept is normally excluded.

Where:

Term	Meaning
Σᵢ(yᵢ − ŷᵢ)²	prediction error
Σⱼβⱼ²	weight-size penalty
λ	regularization strength, called alpha in scikit-learn

The penalty Σⱼβⱼ² is called an L2 penalty.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Simple Ridge Intuition

Imagine two models:

$$\mathrm{A}:\quad\widehat y=2x_1+x_2+0.5x_3$$ $$\mathrm{B}:\quad\widehat y=80x_1-75x_2+20x_3$$

• $x_{1}$: first input feature, shared by both example models.
• $x_{2}$: second input feature, shared by both example models.
• $x_{3}$: third input feature, shared by both example models.
• $\widehat y$ is the prediction; compare coefficient magnitude only after putting features on comparable scales.

Both might fit the training data, but Model B is risky because the weights are huge. Ridge prefers a model like A if both models make similar prediction errors.

So Ridge says:

fit the data, but do not let the weights become unnecessarily large

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

What Alpha Does

The alpha value controls how much Ridge cares about keeping weights small.

alpha value	Behavior
0	Same as ordinary Linear Regression
Small	Light regularization
Medium	Good balance between fit and stability
Very large	Strong shrinkage, possible underfitting

If alpha is too small, Ridge may still overfit. If alpha is too large, the model may become too simple.

Read this lab carefully: polynomial Ridge uses $P=\operatorname{diag}(0,1,\ldots,1)$ so the intercept is unpenalized. The RBF lab fixes a training-mean baseline $\bar y$ and solves for residual targets; prediction is $\bar y+\sum_i\alpha_i K(x,x_i)$. This baseline is not a jointly optimized intercept.

What do regularization and kernel similarity change?

1. Formula and symbols

$$\widehat y(x)=\sum_{j=0}^{p}\beta_jx^j$$

$$(X^TX+\lambda P)\beta=X^Ty,\quad P=\operatorname{diag}(0,1,\ldots,1)$$

$$\mathrm{MSE}=\frac1N\sum_i(\widehat y_i-y_i)^2$$

• $p$: degree
• $X$: polynomial design matrix
• $\beta$: learned coefficients
• $y$: training targets
• $\lambda$: penalty
• $P$: excludes the intercept from that penalty
• $N$: Number of examples in the set being scored

MSE is computed separately on training and validation sets of size N.

Model family

Polynomial RidgeRBF Kernel Ridge

Polynomial degree

5

Regularization log₁₀ λ

-2

Target noise standard deviation

0.2

2. Work with the current inputs

$$(X^TX+1.00\times10^{-2}P)\beta=X^Ty,\quad p=5$$

$$\mathrm{MSE}_{\rm train}=\frac1{12}\sum_i(\widehat y_i-y_i)^2=0.0262$$

$$\mathrm{MSE}_{\rm val}=\frac1{40}\sum_j(\widehat y_j-y_j)^2=0.0711$$

3. Result

Training MSE0.0262

Validation MSE0.0711

Prediction at x = 0.50.991

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Ridge Compared with Other Regressions

Method	Main idea	Best when
Linear Regression	Fit a straight-line relationship	Data is clean and mostly linear
Ridge Regression	Linear Regression with L2 regularization	Features are correlated or noisy
Lasso Regression	Linear Regression with L1 regularization	You want feature selection
Elastic Net	Mixes Ridge and Lasso	You want shrinkage and feature selection
Polynomial Regression	Adds powers like x^2 and x^3	The trend is curved but still simple
SVR	Uses a margin-based regression tube	You want a robust kernel method
Kernel Ridge Regression	Ridge with kernels	The relationship is smooth and non-linear

Ridge is still a linear model. It makes Linear Regression more stable, but it does not automatically learn complex curves unless the features already describe those curves.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Why Ridge Is Useful

Ridge is useful because it improves generalization.

Generalization means:

performing well on new data, not only on the training data

Ridge is a good first choice when:

• You need a strong baseline.
• You want interpretable coefficients.
• Your features are correlated.
• You have many features.
• You suspect ordinary Linear Regression is overfitting.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

But What If the Pattern Is Not Linear?

Linear and Ridge Regression both try to fit a linear relationship in the original feature space.

But many real patterns are not linear.

Example:

x increases -> y first increases, then decreases

A straight line cannot model that well.

One solution is to create new features manually:

original feature: x
new features: x, x^2, x^3

Then a linear model can fit a curve:

$$\widehat y=\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3$$

• $x$: original input feature.
• $x^2$: squared input feature.
• $x^3$: cubed input feature.
• $\beta _{1}$: learned coefficient of the first feature.
• $\beta _{2}$: learned coefficient of the second feature.
• $\beta _{3}$: learned coefficient of the third feature.
• $\beta _{0}$: intercept.

This works, but manually creating all possible useful features can become difficult.

This is where kernels help.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.
• Scikit-learn documentation, "Ridge".

What Is a Kernel?

A kernel is a function that measures similarity between two data points.

Simple idea:

kernel(point A, point B) = how similar A and B are

For a valid real kernel, every Gram matrix is symmetric positive semidefinite. The nearby/high, faraway/low intuition applies to the RBF kernel below; a linear kernel is a dot product and can even be negative.

For example, an RBF kernel behaves like this:

nearby points   -> high similarity
faraway points  -> low similarity

So instead of only asking:

what are the feature values?

a kernel method asks:

how similar is this new point to the training points?

References for this section3

• Scikit-learn, Common pitfalls: data leakage and pipelines.
• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.

Why Do We Use Kernels?

We use kernels because they allow simple algorithms to learn non-linear patterns.

The powerful idea is:

non-linear pattern in original space
        can become
linear pattern in a richer feature space

But we do not want to manually build that richer feature space. A kernel lets the model behave as if it is using many extra transformed features without explicitly creating them.

This is called the kernel trick.

References for this section3

• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.
• Scikit-learn, Common pitfalls: data leakage and pipelines.
• Scikit-learn documentation, "Ridge".

Kernel Trick in Plain Language

Suppose the original data has only two features:

$$x=[x_1,x_2]^T$$

• $x$ is the input vector containing two measured features, x₁ and x₂.

A richer feature space might include:

$$\phi(x)=[x_1,x_2,x_1^2,x_2^2,x_1x_2,\ldots]^T$$

• $\phi (x)$ maps input x to a richer feature vector; products and powers capture nonlinear relationships.

Manually calculating all of these can be expensive.

A kernel gives the model the effect of this richer comparison directly:

compare points in a richer space without explicitly building that space

That is why kernels are useful.

The kernel trick: lift points until a straight line works

1. Formula and symbols

$$\phi(x)=(x,\ x^2),\qquad k(x,z)=\phi(x)^\top\phi(z)=xz+x^2z^2$$

$$\text{lifted rule: }\widehat y=\mathbf1[x^2\ge c]\quad\Leftrightarrow\quad|x|\ge\sqrt c$$

• $x$: Original one-dimensional input
• $\phi(x)$: Feature map: the point drawn in a richer space
• $k(x,z)$: Kernel: the dot product in that space, computed from x and z directly
• $c$: Height of the straight separating line in the lifted space

The kernel trick: an algorithm that only needs dot products can use k(x, z) and never build φ(x) explicitly, even when φ has very many (or infinitely many) dimensions.

Lift amount (pauses the animation)

1

Separating height c

0.25

2. Work with the current inputs

$$\phi(0.8)=(0.8,\ 0.64),\quad\phi(-0.7)=(-0.7,\ 0.49)$$

$$\phi(0.8)^\top\phi(-0.7)=(0.8)(-0.7)+(0.64)(0.49)=-0.2464=k(0.8,-0.7)$$

$$\text{lifted cut }x^2\ge0.25\Rightarrow|x|\ge0.500,\ \text{accuracy}=1.000$$

3. Result

Best single cut on the line80.0%

Horizontal line after lifting100.0%c = 0.25

References for this section3

• Scikit-learn, Common pitfalls: data leakage and pipelines.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.
• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.

Common Kernels

Kernel	Equation	Intuition
Linear	K(x, z) = xᵀz	Ordinary linear similarity
Polynomial	K(x, z) = (xᵀz + c)ᵈ	Captures polynomial curves
RBF	K(x, z) = exp(−γ‖x − z‖²)	Nearby points influence each other strongly
Laplacian	K(x, z) = exp(−γ‖x − z‖)	Similar to RBF, with a different distance shape

Where:

Symbol	Meaning
x	one data point
z	another data point
xᵀz	dot product similarity
‖x − z‖	distance between points
γ	controls how quickly similarity falls with distance
d	polynomial degree

References for this section3

• Scikit-learn, Common pitfalls: data leakage and pipelines.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.
• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.

How Kernel Ridge Regression Works

Kernel Ridge Regression combines two ideas:

Ridge Regression = control model complexity using regularization
Kernel method    = compare points through a similarity function

So Kernel Ridge Regression does this:

1. Measure similarity between training points using a kernel.
2. Learn how much each training point should influence predictions.
3. Regularize the solution so it does not overfit.
4. Predict a new value from similarities to the training data.

The prediction has this form:

$$\widehat y(x_*)=\sum_{i=1}^{n}\alpha_iK(x_*,x_i)$$

• $x_*$ is a new input and xᵢ training input i
• $K$ measures their kernel similarity.
• $\alpha _{i}$ is sample i's learned influence; summing contributions gives prediction ŷ.

Where:

Symbol	Meaning
x*	new point to predict
x₁, x₂, ..., xₙ	training points
K(x*, xᵢ)	similarity between the new point and training point i
αᵢ	learned influence weight for training point i

The training solution can be written compactly as:

$$(K+\lambda I)\alpha=y$$

• $K$ is the training-pair kernel matrix
• $I$ is the identity matrix and λ is regularization strength.
• $y$ is the target vector; solving the linear system yields the influence-weight vector α.

In simple words:

$$\begin{aligned}\widehat y(x_*)={}&\alpha_1K(x_*,x_1)+\alpha_2K(x_*,x_2)\\&+\cdots+\alpha_nK(x_*,x_n)\end{aligned}$$

• $n$ is training-sample count; each αᵢ scales similarity K(x*, xᵢ) to training input xᵢ.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

How the Kernel Helps

Suppose the true pattern is curved.

Plain Ridge tries to fit:

$$\widehat y=\beta_0+\beta^Tx$$

• $\beta ^Tx$ is the weighted sum of feature vector x
• $\beta _{0}$ is the intercept and ŷ the prediction.

Kernel Ridge can fit:

$$\widehat y(x_*)=\sum_{i=1}^{n}\alpha_iK(x_*,x_i)$$

• $K(x_*,x_i)$ compares new input x* with training input xᵢ
• $\alpha _{i}$ is its learned weight.

For RBF kernels, similarity falls with distance. Learned coefficients can have either sign, so the prediction is not simply a normalized average of nearby targets. This combination can fit smooth nonlinear functions.

Kernel ridge regression is a sum of bumps

1. Formula and symbols

$$\widehat y(x)=\bar y+\sum_{i=1}^{n}\alpha_i\,K(x,x_i),\qquad K(x,x_i)=e^{-\gamma(x-x_i)^2}$$

$$(K+\lambda I)\alpha=y-\bar y\mathbf 1$$

• $x_i$: Training input i (12 of them)
• $\alpha_i$: Learned weight of the bump centred on x_i
• $K(x,x_i)$: RBF similarity: 1 at x = x_i, fading with distance
• $\gamma$: Width control: larger γ means narrower bumps
• $\lambda$: Regularisation: larger λ keeps the α small and the curve smooth
• $\bar y$: Training-mean baseline

Every prediction is literally the baseline plus a weighted sum of bumps centred on the training points.

RBF width parameter γ

12

Regularisation log₁₀ λ

λ = 0.100

Highlight bump of training point

6

2. Work with the current inputs

$$\widehat y(0.5)=0.082+1.331-0.663+0.636+\ldots=1.115$$

$$\alpha_{6}=-0.506,\quad x_{6}=-0.091,\quad\max_i|\alpha_i|=1.664$$

3. Result

Training MSE0.0067

Validation MSE0.0883

Largest |αᵢ|1.66big weights mean wiggly fits

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.
• Scikit-learn documentation, "Ridge".

Kernel Ridge Regression vs. KNN Regression

Kernel Ridge Regression and K-Nearest Neighbors can feel similar because both use the training samples during prediction. A new point is compared with stored training points, and nearby or similar points strongly influence the output.

KNN regression predicts by averaging the target values of the nearest neighbors:

$$\widehat y(x_*)=\frac1k\sum_{i\in\mathcal N_k(x_*)}y_i$$

• $k$ is neighbor count
• $\mathcal N_k(x_*)$ is the selected neighbor set for new input x*.
• $y_{i}$ is each neighbor's target; averaging them produces ŷ(x*).

Kernel Ridge Regression predicts using kernel similarities and learned influence weights:

$$\widehat y(x_*)=\sum_{i=1}^{n}\alpha_iK(x_*,x_i)$$

• $\alpha _{i}$ is the learned influence of training input xᵢ
• $K$ compares it with new input x*.

They can perform similarly when:

• The target function is smooth.
• Nearby inputs usually have nearby output values.
• Features are scaled properly.
• The RBF kernel bandwidth and the KNN value of k create a similar neighborhood size.

For an RBF kernel:

$$K(x_*,x_i)=\exp(-\gamma\lVert x_*-x_i\rVert^2)$$

• $\lVert x_*-x_i\rVert^2$ is squared feature-space distance
• $\gamma > 0$ controls how quickly similarity falls with distance.
• $\exp$ is the exponential function
• $K$ approaches 1 for identical inputs and 0 for distant ones.

Nearby points have high similarity, and faraway points have low similarity. This is close in spirit to KNN, where nearby points dominate and faraway points are ignored.

The difference is in how the influence is computed:

Aspect	KNN Regression	Kernel Ridge Regression
Prediction idea	Average nearby target values	Weighted sum of kernel similarities
Neighbor behavior	Uses only the k nearest points	Usually uses all training points, but far points may have tiny kernel values
Weights	Usually equal for the selected neighbors	Learned globally through α = (K + λI)⁻¹y
Training cost	Almost no training	More expensive training because it solves a regularized kernel system
Prediction cost	Can be slow for large training sets	Also can be slow because prediction depends on training samples
Smoothness	Can be piecewise and jumpy	Usually smoother with kernels such as RBF
Regularization	Mainly controlled by k	Controlled by λ and kernel parameters such as γ

So the short intuition is:

KNN: nearby examples vote by their target values.
KRR: similar examples influence the prediction through a smooth, regularized kernel model.

Using training samples in either prediction formula is not data leakage. Fit preprocessing and the model using training observations. Validation may guide tuning, but those observations then cannot serve as an untouched final test. Comparing a test point with stored training inputs at prediction time is legitimate; using test labels or fitting on test observations contaminates final evaluation.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Ridge vs. Kernel Ridge Regression

Question	Ridge Regression	Kernel Ridge Regression
Basic shape	Linear	Non-linear with kernels
Prediction uses	Feature weights	Similarity to training points
Main equation	ŷ = β₀ + βᵀx	ŷ(x*) = ΣᵢαᵢK(x*, xᵢ)
Complexity control	L2 penalty on weights	Ridge-style regularization on kernel solution
Interpretability	Higher	Lower
Training cost	Usually lower	Can be high for large datasets
Prediction speed	Fast	Slower when many training samples exist
Best use	Stable linear baseline	Smooth non-linear pattern

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.

Kernel Ridge vs. Support Vector Regression

Kernel Ridge Regression and Support Vector Regression can both use kernels, but they learn in different ways.

Aspect	Kernel Ridge Regression	Support Vector Regression
Main loss idea	Penalizes squared errors	Ignores small errors inside an epsilon tube
Regularization	Controlled by alpha	Controlled by C
Kernel parameters	Example: gamma for RBF	Example: gamma for RBF
Prediction style	Often uses many training samples	Often uses fewer support vectors
Practical feel	Smooth and conceptually simple	More margin-based and robust

KRR can be easier to understand first because it is basically:

Ridge Regression + kernel similarity

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Which One Should I Use?

Situation	Good starting choice
Need a simple baseline	Linear Regression
Linear model overfits	Ridge Regression
Need feature selection	Lasso Regression
Small dataset where local similarity is enough	KNN Regression
Data has smooth non-linear pattern	Kernel Ridge Regression with RBF
Dataset is very large	Ridge, Random Forest, Gradient Boosting, or neural models
Need very fast prediction	Ridge or another compact linear model
Need a strong tabular model	Random Forest or Gradient Boosting

Start simple. Try Ridge first. If residual plots show curved patterns, then try Kernel Ridge.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn, Common pitfalls: data leakage and pipelines.
• Scikit-learn documentation, "Ridge".

Practical Checklist

• Scale features before using Ridge or KRR.
• Tune alpha with cross-validation.
• For RBF KRR, tune both alpha and gamma.
• Use ordinary Ridge as a baseline before KRR.
• Use KRR carefully on large datasets because kernel matrices can become expensive.
• Check residual plots to see whether the model is missing non-linear structure.

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Python: Ridge Regression

The following examples share the built-in diabetes dataset and train/test split. Grid search wraps the entire scaler/model pipeline so preprocessing is refitted inside each training fold. Lock tuning decisions before inspecting final test scores.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = make_pipeline(
    StandardScaler(),
    Ridge(alpha=1.0)
)

model.fit(X_train, y_train)
pred = model.predict(X_test)

print("MAE:", round(mean_absolute_error(y_test, pred), 3))
print("R2:", round(r2_score(y_test, pred), 3))

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Python: Tune Ridge Alpha

from sklearn.model_selection import GridSearchCV

ridge_cv = GridSearchCV(
    make_pipeline(StandardScaler(), Ridge()),
    param_grid={"ridge__alpha": [0.01, 0.1, 1.0, 10.0, 100.0]},
    scoring="neg_mean_absolute_error",
    cv=5,
)

ridge_cv.fit(X_train, y_train)
pred = ridge_cv.predict(X_test)

print("MAE:", round(mean_absolute_error(y_test, pred), 3))
print("R2:", round(r2_score(y_test, pred), 3))

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn, Common pitfalls: data leakage and pipelines.

Python: Kernel Ridge Regression

from sklearn.kernel_ridge import KernelRidge
from sklearn.model_selection import GridSearchCV

krr = GridSearchCV(
    make_pipeline(StandardScaler(), KernelRidge(kernel="rbf")),
    param_grid={
        "kernelridge__alpha": [0.01, 0.1, 1.0, 10.0],
        "kernelridge__gamma": [0.01, 0.1, 1.0],
    },
    scoring="neg_mean_absolute_error",
    cv=5,
)

krr.fit(X_train, y_train)
pred = krr.predict(X_test)

print("MAE:", round(mean_absolute_error(y_test, pred), 3))
print("R2:", round(r2_score(y_test, pred), 3))

References for this section3

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".
• Scikit-learn documentation, "KernelRidge".

Takeaway

Ridge Regression is Linear Regression with a stabilizing penalty. It is still linear, but it is often more reliable than ordinary Linear Regression when data is noisy or features are correlated.

Kernel Ridge Regression adds kernels, which let the model compare points by similarity and learn smooth non-linear patterns. The easiest way to remember it is:

Ridge Regression controls overfitting.
Kernels add non-linear similarity.
Kernel Ridge Regression combines both.

Continue with Classification, Regression and Compute Cost to build on this note.

References for this section3

• Scikit-learn, Common pitfalls: data leakage and pipelines.
• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• Scikit-learn documentation, "Ridge".