5. Classification Algorithms

What is a Classification Algorithm?

Classification assigns one of several labels to an input.

• Input: tabular row, image, text, signal sample, or measured feature vector.
• Output: a class such as "LOS" or "NLOS", not a distance or power value.
• Evaluation: compare predictions with labels on held-out examples.

Different classifiers make different assumptions. Some draw simple linear boundaries. Some split data into tree-like rules. Some compare a new point with stored examples. Some combine many weak models into a stronger model.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

1. Logistic Regression

Logistic regression is one of the best starting points for binary classification. Despite the word "regression", it is used to estimate the probability of a class.

How does logistic regression learn a straight boundary?

1. Formula and symbols

$$z=w_1x_1+w_2x_2+b,\qquad p=\sigma(z)=\frac{1}{1+e^{-z}}$$

$$L=-\frac1n\sum_{i=1}^{n}\big[y_i\ln p_i+(1-y_i)\ln(1-p_i)\big]$$

$$w_j\leftarrow w_j-\eta\,\frac1n\sum_i(p_i-y_i)x_{ij},\qquad b\leftarrow b-\eta\,\frac1n\sum_i(p_i-y_i)$$

• $x_1$: SNR, rescaled so 0 dB → −1 and 30 dB → +1
• $x_2$: Delay spread, rescaled so 0 ns → −1 and 300 ns → +1
• $w_1,\ w_2$: Learned feature weights
• $b$: Learned bias (intercept)
• $z$: Linear score before the sigmoid
• $p$: Predicted probability that the link is good
• $y_i$: True label of example i (1 = good link, 0 = bad link)
• $L$: Average cross-entropy (log) loss
• $\eta$: Learning rate
• $\tau$: Decision threshold: predict good when p ≥ τ

The boundary p = τ is the straight line z = ln(τ/(1−τ)); logistic regression can only draw straight boundaries in the original features.

Gradient-descent iterations

60

Learning rate η

2

Decision threshold τ

0.5

2. Work with the current inputs

$$w_1=4.99,\quad w_2=-2.81,\quad b=0.05\quad\text{after }60\text{ updates}$$

$$\frac{\partial L}{\partial w}=\big[-0.0161,\ 0.0031\big],\quad\frac{\partial L}{\partial b}=0.0005$$

$$\text{boundary: }4.99x_1-2.81x_2+0.05=\ln\tfrac{0.5}{0.50}=0.00$$

3. Result

Log loss L0.200starts at ln 2 = 0.693

Training accuracy92.5%

Weights w₁ / w₂4.99 / -2.81SNR helps (+), delay hurts (−)

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Intuition

The model computes a weighted sum of the features, then passes it through a sigmoid function. The result is a probability.

$$z=\beta_0+\sum_{j=1}^{p}\beta_jx_j$$

• $z$: score before conversion to a probability.
• $x_{j}$: input feature j.
• $\beta _{j}$: learned weight for feature j.
• $\beta _{0}$: bias or intercept.
• $p$: number of input features.

$$P(y=1\mid x)=\sigma(z)=\frac1{1+e^{-z}}$$

• $P(y=1\mid x)$: probability of class 1 for input x.
• $\sigma$: sigmoid function.
• $z$: weighted model score before the sigmoid.
• $e$: base of the natural exponential.

If the probability is above a chosen threshold, the model predicts class 1; otherwise, class 0.

References for this section3

• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Strengths

• Fast to train.
• Easy to interpret.
• Strong baseline for many problems.
• Works well when the decision boundary is roughly linear.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Weaknesses

• Struggles with complex non-linear patterns unless features are engineered.
• Sensitive to irrelevant features and poor scaling.
• May underfit when the true relationship is highly complex.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Best Use Cases

• Medical risk scoring.
• Spam detection baselines.
• Credit scoring.
• Any problem where interpretability is important.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

2. K-Nearest Neighbors

K-nearest neighbors (KNN) is a memory-based classifier. It does not learn a compact equation during training. Instead, it stores the training examples and compares new examples to them.

Which neighbours actually determine this class?

1. Formula and symbols

$$d_i^2=(x_i-x)^2+(y_i-y)^2$$

$$\widehat c=\mathbf1\!\left[\sum_{i\in\mathcal N_k}c_i>k/2\right]$$

• $x$: First query feature
• $y$: Second query feature
• $x_i$: First feature of training sample i
• $y_i$: Second feature of training sample i
• $d_i$: Euclidean distance
• $c_i$: binary training class
• $\mathcal N_k$: k nearest samples
• $\widehat c$: uniform-majority prediction

Odd k avoids voting ties here.

Query feature x

0.2

Query feature y

0.2

Odd neighbour count k

3

2. Work with the current inputs

$$d_i^2=(x_i-0.20)^2+(y_i-0.20)^2$$

$$\widehat y=\mathbf1\!\left[\sum_{i\in\mathcal N_3}y_i>3/2\right]=0$$

3. Result

Predicted class0

Class-1 neighbour votes1 / 3

Closest squared distance0.044

References for this section3

• Scikit-learn documentation, "LogisticRegression" and "RandomForestClassifier".
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Intuition

To classify a new point:

1. Measure its distance to all training points.
2. Find the K closest points.
3. Take a majority vote among their labels.

If K = 5 and three neighbors are class A while two are class B, the model predicts class A.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Strengths

• Very simple.
• No explicit training phase.
• Can learn non-linear boundaries.
• Useful for small and well-scaled datasets.

References for this section3

• Scikit-learn documentation, "LogisticRegression" and "RandomForestClassifier".
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Weaknesses

• Slow prediction for large datasets.
• Sensitive to feature scaling.
• Performs poorly with many irrelevant features.
• Suffers in high-dimensional spaces.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Best Use Cases

• Small datasets.
• Recommendation prototypes.
• Simple pattern recognition where distance is meaningful.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

3. Decision Trees

A decision tree classifies by asking a sequence of questions. Each internal node tests a feature, each branch is an answer, and each leaf gives a final class.

How does a decision tree carve the feature space?

1. Formula and symbols

$$G(S)=1-p_1^2-p_0^2$$

$$G_{\rm split}=\frac{n_L}{n}G(S_L)+\frac{n_R}{n}G(S_R),\qquad\Delta G=G(S)-G_{\rm split}$$

• $S$: Set of training links that reach a node
• $p_1$: Fraction of good links in S
• $p_0$: Fraction of bad links in S (p₀ = 1 − p₁)
• $G$: Gini impurity: 0 for a pure node, 0.5 for a 50/50 node
• $S_L,\ S_R$: Links sent left (feature ≤ threshold) and right
• $n,\ n_L,\ n_R$: Number of links in S, S_L and S_R
• $\Delta G$: Impurity decrease; the tree picks the split with the largest ΔG

CART tries every feature and every midpoint between sorted values, keeps the best split, then repeats inside each child until the depth limit or a pure node.

Maximum tree depth

3

2. Work with the current inputs

$$G(\text{root})=1-\left(\tfrac{22}{60}\right)^2-\left(\tfrac{38}{60}\right)^2=0.464$$

$$\text{best root split: SNR}\le16.7\text{ dB}\Rightarrow n_L=37,\ n_R=23$$

$$G_{\rm split}=\tfrac{37}{60}(0.149)+\tfrac{23}{60}(0.287)=0.202,\quad\Delta G=0.262$$

3. Result

Leaves (regions)6

Training accuracy98.3%

Agreement with hidden rule88.5%on a 40 × 40 grid

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Intuition

For example, a tree for link quality might ask:

Is SNR > 15 dB?
  yes -> Is delay spread low?
  no  -> Bad link

The tree learns which questions split the data most effectively. Common splitting criteria include Gini impurity and information gain.

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Strengths

• Easy to explain.
• Handles numerical and categorical data.
• Captures non-linear rules.
• Requires little preprocessing.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Weaknesses

• Can overfit badly if allowed to grow too deep.
• Small changes in data can change the tree.
• Single trees are usually less accurate than ensembles.

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Best Use Cases

• Explainable rule-based decisions.
• Quick exploratory modeling.
• Problems where human-readable logic matters.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

4. Support Vector Machines

Support Vector Machines (SVMs) try to find a boundary that separates classes with the largest possible margin.

What does the SVM margin look like, and what does C change?

1. Formula and symbols

$$\min_{w,b,\xi}\ \tfrac12\lVert w\rVert^2+C\sum_{i=1}^{n}\xi_i\quad\text{s.t.}\quad y_i(w^\top x_i+b)\ge1-\xi_i,\ \ \xi_i\ge0$$

$$\text{margin width}=\frac{2}{\lVert w\rVert},\qquad w=\sum_i\alpha_iy_ix_i,\quad0\le\alpha_i\le C$$

• $x_i$: Feature vector of link i (SNR, delay spread), rescaled to [−1, 1]
• $y_i$: Label written as +1 (good) or −1 (bad)
• $w$: Normal vector of the boundary
• $b$: Offset of the boundary
• $\xi_i$: Slack: how far link i is inside the margin or on the wrong side
• $C$: Penalty on slack; large C tolerates fewer margin violations
• $\alpha_i$: Dual weight of link i; links with α > 0 are the support vectors

Solved by dual coordinate descent. The bias is learned as a weight on a constant feature, so it is lightly regularised too.

Penalty log₁₀ C

C = 1.0

2. Work with the current inputs

$$w=(2.36,\ -1.04),\quad b=0.21,\quad\lVert w\rVert=2.576$$

$$\text{margin}=\frac{2}{2.576}=0.776\ \text{(normalised feature units)}$$

$$\#\{\alpha_i>0\}=16,\quad\#\{y_if(x_i)<1\}=14,\quad C=1.0$$

3. Result

Margin width 2/‖w‖0.776

Support vectors16 of 40

Inside margin / misclassified14 / 3

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Intuition

The best boundary is not only one that separates the classes; it is the one that leaves the widest gap between them. The closest training points to the boundary are called support vectors.

For non-linear problems, SVMs use kernels. A kernel lets the algorithm behave as if the data were mapped into a higher-dimensional space, without explicitly computing all those new dimensions.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Strengths

• Works well in high-dimensional data.
• Effective when the number of features is large.
• Kernel trick handles non-linear boundaries.
• Strong theoretical foundation.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Weaknesses

• Slow for very large datasets.
• Hyperparameter tuning is important.
• Results are less interpretable than logistic regression or trees.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Best Use Cases

• Text classification.
• Bioinformatics.
• Medium-sized datasets with many features.
• Problems where a strong margin-based classifier is useful.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

5. Random Forest

Random forest is an ensemble of decision trees. Instead of trusting one tree, it trains many trees and combines their predictions.

Why does a vote of many trees beat one tree?

1. Formula and symbols

$$\mathcal D_t\sim\text{bootstrap}(\mathcal D),\qquad h_t=\text{tree trained on }\mathcal D_t$$

$$v(x)=\frac1T\sum_{t=1}^{T}h_t(x),\qquad\widehat y(x)=\mathbf 1[v(x)\ge0.5]$$

• $\mathcal D$: Original training set of 60 links
• $\mathcal D_t$: Bootstrap sample: 60 draws from D with replacement
• $h_t(x)$: Class (0 or 1) predicted by tree t
• $T$: Number of trees
• $v(x)$: Vote share for the good class
• $\widehat y(x)$: Majority-vote prediction

Each split here may only look at one randomly chosen feature (max_features = 1), which makes the trees disagree; averaging their votes smooths the single tree's jagged regions.

Number of trees T

9

Depth of each tree

3

2. Work with the current inputs

$$v(P)=\frac{1}{9}\sum_t h_t(P)=\frac{5}{9}=0.56\Rightarrow\widehat y(P)=1$$

$$P=(\text{SNR}=18.8\text{ dB},\ \text{delay}=165\text{ ns})$$

3. Result

Rule agreement, forest87.9%one tree alone: 61.7%

Training accuracy96.7%

Vote share at P5 / 9 trees

References for this section3

• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.

Intuition

Each tree sees a random sample of the data and a random subset of features. Because the trees are different, their errors are partly different. Voting across many trees reduces the instability of a single tree.

References for this section3

• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.

Strengths

• Strong performance on tabular data.
• Reduces decision-tree overfitting.
• Handles non-linear feature interactions.
• Provides feature importance estimates.
• Works with little preprocessing.

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Weaknesses

• Less interpretable than a single tree.
• Larger memory usage.
• Cannot extrapolate beyond patterns seen in training.

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Best Use Cases

• General tabular classification.
• Baselines for structured data.
• Problems with mixed feature types.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

6. Gradient Boosting and XGBoost

Gradient boosting also uses many trees, but it builds them sequentially. Each new tree tries to correct the errors made by the previous trees.

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.

Intuition

Random forest builds many independent trees and averages them. Boosting builds a team in order: tree 1 makes mistakes, tree 2 focuses on those mistakes, tree 3 corrects remaining errors, and so on.

XGBoost is a highly optimized implementation of gradient-boosted trees. It became popular because it is accurate, regularized, scalable, and very effective on structured data.

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Strengths

• Excellent accuracy on tabular data.
• Handles non-linear interactions.
• Includes regularization to control overfitting.
• Often performs better than random forest when tuned carefully.

References for this section3

• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Weaknesses

• More hyperparameters to tune.
• Can overfit if trees are too deep or learning rate is too high.
• Less transparent than simple models.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Best Use Cases

• Structured/tabular datasets.
• Competitions and high-performance prediction tasks.
• Risk scoring, churn prediction, demand forecasting, and ranking tasks.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

7. Neural Network Classifiers

Neural networks are useful when the input is complex, high-dimensional, or weakly structured, such as images, audio, text, or raw sensor streams.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Intuition

A neural network learns layers of representation. Early layers learn simple patterns, while deeper layers combine them into more abstract concepts.

References for this section3

• Scikit-learn documentation, "LogisticRegression" and "RandomForestClassifier".
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Strengths

• Powerful for images, language, speech, and large datasets.
• Learns features automatically.
• Can model highly complex boundaries.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Weaknesses

• Requires more data and compute.
• Harder to interpret.
• More sensitive to architecture and training choices.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Model Selection Guide

Need	Good starting model
Simple, interpretable baseline	Logistic regression
Small dataset with meaningful distance	KNN
Human-readable rules	Decision tree
Strong classifier for many features	SVM
Reliable tabular baseline	Random forest
High-performance tabular prediction	XGBoost or gradient boosting
Images, speech, text, raw signals	Neural network

Same data, six classifiers: which boundary fits the moons?

1. Formula and symbols

$$\mathrm{Acc}=\frac1N\sum_{i=1}^{N}\mathbf 1[\widehat y_i=y_i]$$

$$\text{gap}=\mathrm{Acc}_{\rm train}-\mathrm{Acc}_{\rm val}$$

• $N$: Number of examples scored (80 training or 200 validation)
• $\widehat y_i$: Predicted class of example i
• $y_i$: True class of example i
• $\mathbf 1[\cdot]$: 1 when the condition holds, else 0
• $\text{gap}$: How much better the model does on data it has seen

All six models train on the same 80 points and are scored on 200 separate validation points from the same two-moons distribution.

Classifier

LogisticKNN (k=7)Tree (depth 5)Forest (25)RBF SVMNeural net

2. Work with the current inputs

$$\widehat y=\mathbf1\big[\textstyle\sum_i\alpha_iy_i(e^{-4\lVert x-x_i\rVert^2}+1)\ge0\big],\ C=3$$

$$\mathrm{Acc}_{\rm train}=1.000,\quad\mathrm{Acc}_{\rm val}=\tfrac{198}{200}=0.990,\quad\text{gap}=0.010$$

3. Result

RBF SVM validation accuracy99.0%

Training accuracy100.0%

Best on validationKNN (k=7)

References for this section3

• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.

Evaluation Checklist

• Use a separate test set.
• Check class imbalance.
• Report precision, recall, F1-score, and confusion matrix.
• Tune thresholds when false positives and false negatives have different costs.
• Use cross-validation when data is limited.
• Inspect failure cases, not only summary metrics.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Python: Load Common Classifiers

All of these scikit-learn classifiers use the same core methods:

• fit(X_train, y_train) learns from data.
• predict(X_test) returns class labels.
• predict_proba(X_test) returns probabilities when supported.

from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.naive_bayes import GaussianNB

classifiers = {
    "logistic_regression": LogisticRegression(max_iter=1000),
    "knn": KNeighborsClassifier(n_neighbors=5),
    "decision_tree": DecisionTreeClassifier(max_depth=5, random_state=42),
    "svm": SVC(kernel="rbf", probability=True),
    "random_forest": RandomForestClassifier(n_estimators=200, random_state=42),
    "gradient_boosting": GradientBoostingClassifier(random_state=42),
    "naive_bayes": GaussianNB(),
}

References for this section3

• Scikit-learn documentation, "LogisticRegression" and "RandomForestClassifier".
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.
• T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System", Proceedings of ACM SIGKDD, San Francisco, CA, USA, 2016, pp. 785-794.

Python: Compare Classifiers on One Dataset

Select the family on validation data, refit that choice on development data, then inspect the final test score once. This small example uses a single split; use fold-local pipelines for cross-validation.

from sklearn.base import clone
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, f1_score

X, y = load_breast_cancer(return_X_y=True)
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
X_train, X_validation, y_train, y_validation = train_test_split(
    X_dev, y_dev, test_size=0.25, random_state=43, stratify=y_dev
)

scores = {}
for name, clf in classifiers.items():
    candidate = make_pipeline(StandardScaler(), clone(clf))
    candidate.fit(X_train, y_train)
    pred = candidate.predict(X_validation)
    scores[name] = f1_score(y_validation, pred)
    print(name, "validation F1=", round(scores[name], 3))

best_name = max(scores, key=scores.get)
model = make_pipeline(StandardScaler(), clone(classifiers[best_name]))
model.fit(X_dev, y_dev)
pred = model.predict(X_test)
print("Selected:", best_name)
print("Final test accuracy:", accuracy_score(y_test, pred))
print("Final test F1:", f1_score(y_test, pred))

References for this section3

• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.

Python: Save and Load a Trained Classifier

import joblib

best_model = make_pipeline(
    StandardScaler(),
    RandomForestClassifier(n_estimators=300, random_state=42)
)
best_model.fit(X_train, y_train)

joblib.dump(best_model, "classifier.joblib")

loaded_model = joblib.load("classifier.joblib")
print(loaded_model.predict(X_test[:5]))

References for this section3

• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• Scikit-learn documentation, "LogisticRegression" and "RandomForestClassifier".
• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.

Takeaway

No classifier is best for every problem. Start with a simple baseline, understand the data, choose metrics that match the real cost of mistakes, and increase model complexity only when the simpler model is not enough.

Continue with Regression Algorithms to build on this note.

References for this section3

• C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
• L. Breiman, "Random Forests", Machine Learning, vol. 45, pp. 5-32, 2001.
• J. H. Friedman, "Greedy Function Approximation: A Gradient Boosting Machine", The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.