8. Classification, Regression and Compute Cost

The Core Difference

Classifiers and regressors are both supervised learning models. The difference is the type of target they predict.

Question	Model type	Output
Is this sample LOS or NLOS?	Classifier	Class label
What is the received signal strength?	Regressor	Continuous number
Which disease category is present?	Classifier	Class label
What is the expected localization error?	Regressor	Continuous number

A classifier predicts categories. A regressor predicts quantities.

References for this section3

• Scikit-learn documentation, "Supervised learning".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Classifier Outputs

A classifier usually returns one or both of these:

• A class label, such as 0, 1, "normal", or "fault".
• A class probability, such as P(class = fault) = 0.82.

The decision can depend on a threshold. For example, a binary classifier may predict class 1 when the probability is above 0.5, but a high-risk system may use a lower or higher threshold depending on the cost of mistakes.

References for this section3

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• Scikit-learn documentation, "Supervised learning".

Regressor Outputs

A regressor returns a numerical value:

predicted house price = 315000
predicted temperature = 21.4
predicted positioning error = 0.73 meters

The output is not a class. It is a value on a continuous scale.

References for this section3

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• Scikit-learn documentation, "Supervised learning".

Evaluation Metrics

Classifiers and regressors need different metrics.

Goal	Classifier metrics	Regressor metrics
General performance	Accuracy, balanced accuracy	MAE, RMSE, R-squared
Imbalanced data	Precision, recall, F1-score, ROC-AUC	Not directly applicable
Error cost	Confusion matrix, false-positive rate, false-negative rate	Residual plots, absolute error, squared error
Interpretability	Feature importance, coefficients, decision rules	Feature importance, coefficients, residual behavior

Accuracy is not meaningful for continuous regression. RMSE is not meaningful for class labels unless the labels have a real numeric distance.

$$\mathrm{Accuracy}=\frac{\mathrm{correct\ predictions}}{\mathrm{total\ predictions}}$$ $$F_1=\frac{2\,\mathrm{Precision}\,\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

• Accuracy: fraction of all predictions that are correct.
• Precision: true positives divided by predicted positives
• Recall: true positives divided by actual positives.
• F1: harmonic mean of precision and recall; useful when both error types matter.

$$\mathrm{MAE}=\frac1n\sum_{i=1}^n\lvert y_i-\widehat y_i\rvert$$ $$\mathrm{RMSE}=\sqrt{\frac1n\sum_{i=1}^n(y_i-\widehat y_i)^2}$$

• $n$ is the number of samples
• $y_{i}$ is a measured value and ŷᵢ its prediction.
• $\sum _{i}$ adds errors across samples; absolute value gives MAE, while squaring then taking the square root gives RMSE.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Efficiency Means More Than Accuracy

Efficiency can mean several things:

• Training time.
• Prediction latency.
• Memory usage.
• Amount of preprocessing.
• Amount of labeled data needed.
• Ease of tuning.
• Interpretability per unit of complexity.

A model with slightly lower accuracy may be better if it is faster, easier to maintain, and more stable.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Computational Efficiency by Model Family

Model family	Classifier example	Regressor example	Efficiency notes
Linear models	Logistic Regression	Linear/Ridge Regression	Fast, scalable, interpretable
KNN	KNeighborsClassifier	KNeighborsRegressor	Cheap training, slow prediction on large data
Trees	DecisionTreeClassifier	DecisionTreeRegressor	Fast, interpretable, can overfit
Random forests	RandomForestClassifier	RandomForestRegressor	Strong but heavier than one tree
Boosting	GradientBoostingClassifier	GradientBoostingRegressor	Accurate, needs tuning, sequential training
Kernel methods	SVC	SVR/Kernel Ridge	Powerful but expensive for large datasets
Neural networks	MLP/CNN classifier	MLP regressor	Efficient at scale with hardware, data-hungry

References for this section3

• Scikit-learn documentation, "Model evaluation".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Which Is Usually Faster?

There is no universal winner. The task type does not determine speed by itself; the algorithm and dataset do.

For example:

• Logistic Regression and Ridge Regression are both usually fast.
• KNN classification and KNN regression both become slow at prediction time as the training set grows.
• Random Forest classifiers and regressors have similar computational patterns.
• Kernel classifiers and kernel regressors can both be expensive on large datasets.
• Neural classifiers and neural regressors can both require significant compute.

The model family matters more than whether the task is classification or regression.

References for this section3

• Scikit-learn documentation, "Model evaluation".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Data Efficiency

Data efficiency asks how much data a model needs before it performs well.

Situation	More data-efficient choices
Small tabular dataset	Linear models, Ridge, Logistic Regression, trees
Strong domain features	Linear models and tree ensembles
Complex images/audio/text	Neural networks, usually with transfer learning
Smooth non-linear data	Kernel methods or gradient boosting
Noisy measurements	Regularized models and ensembles

Regularization improves data efficiency because it reduces overfitting.

References for this section3

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• Scikit-learn documentation, "Supervised learning".

Prediction Efficiency

Prediction efficiency matters when a model runs in real time, on embedded hardware, or inside a large simulation loop.

Model	Prediction behavior
Linear/Logistic/Ridge	Very fast matrix multiplication
Decision tree	Fast path through tree nodes
Random forest	Slower because many trees vote or average
Gradient boosting	Slower than one tree, often faster than large forests
KNN	Can be slow because it compares against stored samples
Kernel Ridge	Can be slow because prediction depends on training examples
Neural network	Fast on GPUs, may be heavy on small CPUs

What grows when a model keeps every training example?

1. Formula and symbols

$$N_{\rm linear}=d+1,\quad C_{\rm KNN}\sim nd$$

$$M_{\rm kernel}=8n^2\ \mathrm{bytes}$$

• $d$: features
• $n$: stored training examples

The linear count includes an intercept

KNN counts coordinate comparisons

the kernel matrix uses 8-byte values. These different size proxies are not runtime measurements.

Stored training examples n

1000

Input features d

20

2. Work with the current inputs

$$C_{\rm linear}\sim d=20,\quad C_{\rm KNN}\sim nd=1000\times20=20000$$

$$M_K=8n^2=8\times1000^2\ \mathrm{bytes}=7.63\ \mathrm{MiB}$$

3. Result

Linear coefficient slots21

Brute-force KNN coordinate comparisons20,000

Dense KRR matrix only7.63 MiB

References for this section3

• Scikit-learn documentation, "Model evaluation".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Choosing Between Classification and Regression

Choose classification when the real target is a category:

• Fault type.
• Link state.
• Object class.
• User activity.
• Disease category.

Choose regression when the real target is a quantity:

• Distance.
• Power.
• Temperature.
• Price.
• Latency.
• Error magnitude.

If a numeric value is later converted into categories, regression may preserve more information. If only the category matters, classification is usually simpler.

References for this section3

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• Scikit-learn documentation, "Supervised learning".

Borderline Cases

Some problems can be framed either way.

Problem	Classification framing	Regression framing
Signal quality	poor / fair / good	SINR value
Risk	low / medium / high	probability or score
Localization	room ID	x-y coordinates
Demand	low / normal / high	number of units

The better framing depends on the decision that follows the prediction.

References for this section3

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• Scikit-learn documentation, "Supervised learning".

Practical Selection Guide

Need	Recommended direction
Need exact numerical estimate	Regression
Need category or action label	Classification
Need uncertainty over classes	Probabilistic classification
Need simple and fast baseline	Logistic Regression or Ridge Regression
Need high tabular accuracy	Gradient boosting or random forest
Need real-time prediction	Linear model, small tree, or compact neural model
Need interpretability	Linear model, shallow tree, or feature importance analysis

References for this section3

• Scikit-learn documentation, "Model evaluation".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Python: Compare Classifier and Regressor Workflows

from sklearn.datasets import load_breast_cancer, fetch_california_housing
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.metrics import accuracy_score, mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Classification task
X_cls, y_cls = load_breast_cancer(return_X_y=True)
Xc_train, Xc_test, yc_train, yc_test = train_test_split(
    X_cls, y_cls, test_size=0.2, random_state=42, stratify=y_cls
)

classifier = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=1000)
)
classifier.fit(Xc_train, yc_train)
cls_pred = classifier.predict(Xc_test)

print("classification accuracy:", accuracy_score(yc_test, cls_pred))

# Regression task
X_reg, y_reg = fetch_california_housing(return_X_y=True)
Xr_train, Xr_test, yr_train, yr_test = train_test_split(
    X_reg, y_reg, test_size=0.2, random_state=42
)

regressor = make_pipeline(
    StandardScaler(),
    Ridge(alpha=1.0)
)
regressor.fit(Xr_train, yr_train)
reg_pred = regressor.predict(Xr_test)

print("regression MAE:", mean_absolute_error(yr_test, reg_pred))

References for this section3

• Scikit-learn documentation, "Model evaluation".
• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

Takeaway

• Classifier: answers "which category?"; evaluate with class-relevant metrics.
• Regressor: answers "how much?"; evaluate with numeric-error metrics.
• Efficiency: depends on dataset size, feature count, algorithm, and deployment budget.
• First decision: choose the right target type, then compare meaningful errors and compute cost.

Continue with Deep Learning Architectures to build on this note.

References for this section3

• T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed., Springer, 2009.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• Scikit-learn documentation, "Supervised learning".