6. Regression Algorithms

What is Regression?

Regression is a supervised learning task where the target is a continuous numerical value. Instead of predicting a class such as "spam" or "not spam", a regression model predicts a quantity.

Examples:

• House price in pounds.
• Temperature tomorrow.
• Received signal strength.
• Localization error in meters.
• Battery lifetime in hours.

The central question is: "Given these input features, what number should the model predict?"

Regression vs. Classification

1. Linear Regression

Linear regression is the simplest and most interpretable regression model. It assumes the output can be approximated by a weighted sum of the input features.

Intuition

For one feature:

• $\widehat y$ is the predicted output
• $x$: input feature.
• $\beta _{1}$: learned slope.
• $\beta _{0}$: intercept.

For many features:

• $x_{j}$: input feature j.
• $\beta _{j}$: learned weight for feature j.
• $p$: number of features.
• $\beta _{0}$ remains the intercept.

Training finds weights that minimize the difference between predicted and true values, often using mean squared error.

Strengths

• Very fast.
• Easy to interpret.
• Good baseline.
• Can extrapolate outside the training range.

Weaknesses

• Assumes a mostly linear relationship.
• Sensitive to outliers.
• Cannot naturally model complex feature interactions.

Best Use Cases

• Quick baselines.
• Scientific modeling where interpretability matters.
• Problems with approximately linear relationships.

2. Ridge and Lasso Regression

Ridge and Lasso are regularized versions of linear regression. They add a penalty to discourage overly large weights.

Ridge Regression

Ridge uses an L2 penalty. It shrinks weights smoothly and is useful when many features are correlated.

Equation-style view:

• $y_{i}$: measured target for sample i.
• $\widehat y_i$: model prediction for sample i.
• $\sum _{i}$ adds squared errors across samples.

• $\sum _{i}(y_{i} - \widehat y_{i})^2$ measures fit error
• $\lambda \sum _{j}\beta _{j}^2$ penalizes large feature weights.
• $\lambda ≥ 0$ controls penalty strength; the intercept is normally excluded from the penalty.

In simple words, Ridge fits the data while keeping the weights from becoming unnecessarily large. This often makes the model more stable on new data.

Lasso Regression

Lasso uses an L1 penalty. It can push some weights exactly to zero, which makes it useful for feature selection.

• $\sum _{i}(y_{i} - \widehat y_{i})^2$ measures fit error
• $\lambda \sum _{j}|\beta _{j}|$ penalizes absolute feature weights.
• $\lambda ≥ 0$ controls penalty strength; the intercept is normally excluded.

Why Regularization Matters

Without regularization, a model may fit noise in the training data. Regularization constrains the fit and can improve generalization; select its strength using validation.

3. K-Nearest Neighbors Regressor

KNN regression predicts a number by averaging the target values of nearby training examples.

Intuition

If a new house is similar to five nearby houses in the dataset, KNN predicts the average price of those five houses.

Strengths

• Simple and non-parametric.
• Can model non-linear relationships.
• No explicit training phase.

Weaknesses

• Slow prediction for large datasets.
• Sensitive to feature scaling.
• Performs poorly when irrelevant features dominate distance.

Best Use Cases

• Small datasets.
• Local interpolation problems.
• Situations where similar inputs should have similar outputs.

4. Polynomial Regression

Polynomial regression extends linear regression by adding powers of features, such as x^2, x^3, and interactions.

Intuition

A straight line cannot fit a curved trend. Polynomial features allow a linear model to fit curves.

• $x$: original input feature.
• $x^2$: squared input feature.
• $x^3$: cubed input feature.
• $\beta _{1}$: learned coefficient of the first feature.
• $\beta _{2}$: learned coefficient of the second feature.
• $\beta _{3}$: learned coefficient of the third feature.
• $\beta _{0}$: intercept.

Strengths

• Captures simple curves.
• Still relatively interpretable.
• Easy to implement.

Weaknesses

• High-degree polynomials can overfit badly.
• Extrapolation can become unrealistic.
• Feature scaling becomes important.

5. Support Vector Regression

Support Vector Regression (SVR) is related to SVM classification. Instead of separating classes, it tries to fit a function while ignoring small errors inside a margin.

Intuition

SVR draws a tube around the prediction function. Errors inside the tube are acceptable. The model focuses mainly on points outside the tube.

Strengths

• Robust to small noise.
• Kernel functions allow non-linear regression.
• Useful for medium-sized datasets.

Weaknesses

• Can be slow for large datasets.
• Requires tuning kernel, margin, and regularization parameters.
• Less interpretable than linear regression.

6. Decision Tree Regressor

A decision tree regressor splits the feature space into regions and predicts the average target value in each region.

Intuition

For house prices, a tree may first split by location, then by area, then by number of rooms. Each final leaf stores an average price.

Strengths

• Handles non-linear relationships.
• Easy to visualize.
• Minimal preprocessing.

Weaknesses

• Can overfit if too deep.
• Predictions are piecewise constant.
• Single trees are unstable.

7. Random Forest Regressor

Random forest regression averages predictions from many decision trees.

Intuition

Each tree makes a different estimate. Averaging many trees reduces variance and gives a more stable prediction.

Strengths

• Strong tabular-data baseline.
• Handles feature interactions.
• Less overfitting than one decision tree.
• Works with minimal preprocessing.

Weaknesses

• Less interpretable than one tree.
• Cannot extrapolate beyond the training target range.
• Larger memory footprint.

8. Gradient Boosting and XGBoost Regressor

Gradient boosting builds trees sequentially. Each new tree learns to correct the residual errors of the current model.

Intuition

If the model underpredicts one group of samples, the next tree focuses on reducing that underprediction. Over many rounds, the model becomes a strong predictor.

Strengths

• Often excellent on structured data.
• Captures non-linear relationships.
• Works well with mixed feature types.
• XGBoost adds regularization and scalable training.

Weaknesses

• Needs careful tuning.
• Can overfit if trees are too deep or learning rate is too large.
• Training is more sequential than random forest.

Regression Metrics

Mean Absolute Error

Mean Absolute Error (MAE) is the average absolute difference between prediction and true value. It is easy to interpret because it uses the same unit as the target.

• $n$ is sample count
• $y_{i} - \widehat y_{i}$ is sample i's prediction error; absolute value prevents opposite errors cancelling.

Mean Squared Error

Mean Squared Error (MSE) squares the errors before averaging. Large errors are punished more strongly.

• $n$ is sample count
• $y_{i}$ is measured and ŷᵢ predicted; squaring emphasizes larger errors.

Root Mean Squared Error

Root Mean Squared Error (RMSE) is the square root of MSE. It is also in the same unit as the target.

• $n$ is sample count
• $y_{i} - \widehat y_{i}$ is each error; the square root returns the metric to the target's unit.

R-squared

R-squared measures how much variance in the target is explained by the model. It is useful, but it should not be the only metric.

• $y_{i}$: measured target for sample i.
• $\widehat y_i$: prediction for sample i.
• $\bar y$: mean measured target over the evaluation set.
• The numerator is model squared error; the denominator is squared error from always predicting the mean.

Practical Checklist

• Plot predicted values vs. true values.
• Check residuals for patterns.
• Scale features for KNN, SVR, linear models, and neural networks.
• Use cross-validation when data is limited.
• Compare against a simple baseline, such as predicting the mean.
• Watch out for outliers, data leakage, and target leakage.

Python: Load Common Regressors

Regressors follow the same scikit-learn pattern as classifiers:

• fit(X_train, y_train) learns the mapping.
• predict(X_test) returns continuous values.
• Metrics such as MAE, RMSE, and R-squared evaluate the predictions.

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor

regressors = {
    "linear": LinearRegression(),
    "ridge": Ridge(alpha=1.0),
    "lasso": Lasso(alpha=0.01),
    "knn": KNeighborsRegressor(n_neighbors=5),
    "decision_tree": DecisionTreeRegressor(max_depth=6, random_state=42),
    "svr": SVR(kernel="rbf"),
    "random_forest": RandomForestRegressor(n_estimators=200, random_state=42),
    "gradient_boosting": GradientBoostingRegressor(random_state=42),
}

Python: Compare Regressors on One Dataset

Select the family on validation data, refit that choice on development data, then inspect the final test score once. This small example uses a single split; use fold-local pipelines for cross-validation.

from sklearn.base import clone
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error, r2_score

X, y = load_diabetes(return_X_y=True)
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_train, X_validation, y_train, y_validation = train_test_split(
    X_dev, y_dev, test_size=0.25, random_state=43
)

scores = {}
for name, reg in regressors.items():
    candidate = make_pipeline(StandardScaler(), clone(reg))
    candidate.fit(X_train, y_train)
    pred = candidate.predict(X_validation)
    scores[name] = mean_absolute_error(y_validation, pred)
    print(name, "validation MAE=", round(scores[name], 3))

best_name = min(scores, key=scores.get)
model = make_pipeline(StandardScaler(), clone(regressors[best_name]))
model.fit(X_dev, y_dev)
pred = model.predict(X_test)
print("Selected:", best_name)
print("Final test MAE:", mean_absolute_error(y_test, pred))
print("Final test R2:", r2_score(y_test, pred))

Python: Save and Load a Regressor

import joblib

final_regressor = RandomForestRegressor(
    n_estimators=300,
    max_depth=None,
    random_state=42,
    n_jobs=-1
)
final_regressor.fit(X_train, y_train)

joblib.dump(final_regressor, "regressor.joblib")

loaded_regressor = joblib.load("regressor.joblib")
print(loaded_regressor.predict(X_test[:5]))

Takeaway

• Regression output: a number, such as distance or received power.
• Linear model: fast and interpretable.
• Tree ensemble: often strong on tabular features; kernel/neural model: can learn more complex nonlinear patterns.
• Choose by: data shape, interpretability, error cost, and deployment budget.

Continue with Ridge and Kernel Ridge Regression to build on this note.