Classification Algorithms

Decision Trees are intuitive, interpretable models that make decisions by asking a series of yes/no questions. Random Forests combine multiple trees to create a powerful, robust ensemble that often outperforms individual models.

How Decision Trees Work

Imagine you're trying to decide if someone will like a movie. You might ask: "Do they like action movies?" If yes, you ask: "Are they over 25?" If yes again, you predict they'll like it. This is exactly how a decision tree works - it asks a series of yes/no questions to reach a decision.

A decision tree splits data based on feature values, creating a tree-like structure. At each node (decision point), it asks a question like "Is age > 30?" or "Is income > $50,000?" and branches left or right accordingly. It keeps asking questions until it reaches a leaf node (final answer) where it makes a prediction. The tree automatically learns which questions to ask and in what order by analyzing the training data!

Decision Tree Implementation

STEP 1: Load Data and Train Decision Tree

# Decision Tree for Classification
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# Load Iris dataset
iris = load_iris()
X, y = iris.data, iris.target
feature_names = iris.feature_names
class_names = iris.target_names

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Create and train Decision Tree
tree_model = DecisionTreeClassifier(
    max_depth=3,           # Limit depth to prevent overfitting
    criterion='gini',      # 'gini' or 'entropy'
    random_state=42
)
tree_model.fit(X_train, y_train)

print(f"Training Accuracy: {tree_model.score(X_train, y_train):.2%}")
print(f"Test Accuracy: {tree_model.score(X_test, y_test):.2%}")

We limit max_depth=3 to prevent overfitting. The tree learns rules like "if petal_width ≤ 0.8 then Setosa" by choosing splits that maximize information gain or minimize Gini impurity at each node.

STEP 2: Visualize the Tree Structure

# Visualize the Decision Tree
plt.figure(figsize=(20, 10))
plot_tree(
    tree_model,
    feature_names=feature_names,
    class_names=class_names,
    filled=True,
    rounded=True,
    fontsize=10
)
plt.title("Decision Tree for Iris Classification")
plt.tight_layout()
plt.show()

# Feature importance - which features matter most?
importance = pd.DataFrame({
    'Feature': feature_names,
    'Importance': tree_model.feature_importances_
}).sort_values('Importance', ascending=False)

print("Feature Importance:")
print(importance.to_string(index=False))

Random Forests: The Power of Ensembles

A Random Forest is like asking 100 different experts for their opinion and taking a vote, instead of trusting just one expert. It builds many decision trees (often 100-500 trees) and combines their predictions. Each tree is trained on a different random subset of data (called bagging or bootstrap sampling) and considers different random subsets of features at each split. This randomness makes each tree unique and prevents them from all making the same mistakes.

# Random Forest Classifier
from sklearn.ensemble import RandomForestClassifier

# Create Random Forest with 100 trees
rf_model = RandomForestClassifier(
    n_estimators=100,      # Number of trees
    max_depth=5,           # Depth per tree
    min_samples_split=5,   # Min samples to split
    random_state=42,
    n_jobs=-1              # Use all CPU cores
)

rf_model.fit(X_train, y_train)

print(f"Random Forest Training Accuracy: {rf_model.score(X_train, y_train):.2%}")
print(f"Random Forest Test Accuracy: {rf_model.score(X_test, y_test):.2%}")

# Compare single tree vs Random Forest
from sklearn.metrics import classification_report

# Single tree predictions
tree_pred = tree_model.predict(X_test)
# Random Forest predictions
rf_pred = rf_model.predict(X_test)

print("=== Single Decision Tree ===")
print(classification_report(y_test, tree_pred, target_names=class_names))

print("\n=== Random Forest (100 Trees) ===")
print(classification_report(y_test, rf_pred, target_names=class_names))

# Random Forest Feature Importance
rf_importance = pd.DataFrame({
    'Feature': feature_names,
    'Importance': rf_model.feature_importances_
}).sort_values('Importance', ascending=False)

print("Random Forest Feature Importance:")
print(rf_importance.to_string(index=False))

# Random Forest importance is more reliable than single tree
# because it's averaged across many trees

Key Hyperparameters

# STEP 1: Load data and create decision tree
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import pandas as pd

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# STEP 2: Train decision tree with limited depth
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
print(f"Test Accuracy: {tree.score(X_test, y_test):.2%}")

# STEP 3: Extract feature importances
importances = pd.DataFrame({
    'Feature': iris.feature_names,
    'Importance': tree.feature_importances_
}).sort_values('Importance', ascending=False)
print("\nFeature Importances:")
print(importances.to_string(index=False))

# STEP 4: Understand the output
print("\nThe most important feature has the highest impact on predictions!")

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

# STEP 1: Define parameter grid
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 7, 10],
    'min_samples_split': [2, 5, 10]
}

# STEP 2: Perform grid search
grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,  # Use all CPU cores
    verbose=1
)
grid_search.fit(X_train, y_train)

# STEP 3: Display results
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.2%}")
print(f"Test score: {grid_search.score(X_test, y_test):.2%}")

# STEP 4: Get feature importances from best model
best_rf = grid_search.best_estimator_
feature_imp = pd.DataFrame({
    'Feature': iris.feature_names,
    'Importance': best_rf.feature_importances_
}).sort_values('Importance', ascending=False)
print("\nOptimal Random Forest Feature Importances:")
print(feature_imp.to_string(index=False))

# STEP 1: Train decision tree without depth limit (will overfit)
tree_overfit = DecisionTreeClassifier(random_state=42)  # No max_depth
tree_overfit.fit(X_train, y_train)

# STEP 2: Train random forest
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

# STEP 3: Compare training vs test accuracy
print("=== Single Decision Tree ===")
print(f"Training Accuracy: {tree_overfit.score(X_train, y_train):.2%}")
print(f"Test Accuracy: {tree_overfit.score(X_test, y_test):.2%}")
print(f"Overfitting gap: {tree_overfit.score(X_train, y_train) - tree_overfit.score(X_test, y_test):.2%}")

print("\n=== Random Forest ===")
print(f"Training Accuracy: {rf.score(X_train, y_train):.2%}")
print(f"Test Accuracy: {rf.score(X_test, y_test):.2%}")
print(f"Overfitting gap: {rf.score(X_train, y_train) - rf.score(X_test, y_test):.2%}")

print("\nInsight: Random Forest has smaller train-test gap = less overfitting!")

import matplotlib.pyplot as plt

# STEP 1: Get pruning path
tree_full = DecisionTreeClassifier(random_state=42)
path = tree_full.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas[:-1]  # Remove max alpha

# STEP 2: Train trees with different alpha values
train_scores = []
test_scores = []

for ccp_alpha in ccp_alphas:
    tree = DecisionTreeClassifier(ccp_alpha=ccp_alpha, random_state=42)
    tree.fit(X_train, y_train)
    train_scores.append(tree.score(X_train, y_train))
    test_scores.append(tree.score(X_test, y_test))

# STEP 3: Find optimal alpha
best_idx = test_scores.index(max(test_scores))
best_alpha = ccp_alphas[best_idx]
print(f"Optimal ccp_alpha: {best_alpha:.4f}")
print(f"Best test accuracy: {max(test_scores):.2%}")

# STEP 4: Visualize pruning effect
plt.figure(figsize=(10, 5))
plt.plot(ccp_alphas, train_scores, marker='o', label='Train', drawstyle="steps-post")
plt.plot(ccp_alphas, test_scores, marker='o', label='Test', drawstyle="steps-post")
plt.axvline(best_alpha, color='r', linestyle='--', label=f'Best alpha={best_alpha:.4f}')
plt.xlabel('ccp_alpha (complexity parameter)')
plt.ylabel('Accuracy')
plt.title('Pruning Path: Finding Optimal Tree Complexity')
plt.legend()
plt.grid(True)
plt.show()

print("\nAs alpha increases, tree becomes simpler (less overfitting but may underfit)")

SVM is a powerful classification algorithm that finds the optimal hyperplane separating different classes. It excels in high-dimensional spaces and can handle non-linear boundaries using the "kernel trick."

The Intuition Behind SVM

Imagine you have red dots and blue dots on a piece of paper, and you need to draw a line to separate them. There are many lines you could draw, but SVM finds the best line - the one with the most "breathing room" on both sides. This "breathing room" is called the margin.

Think of it like drawing a road between two neighborhoods. SVM makes the road as wide as possible while still separating the neighborhoods. The houses closest to the road (called support vectors) determine where the road goes - that's why they're so important! In higher dimensions (more than 2 features), the "line" becomes a hyperplane, but the concept is the same: maximize the margin for better generalization.

Linear SVM Implementation

Let's start with a simple linear SVM to understand the basics. We'll generate synthetic data that's linearly separable (can be divided by a straight line).

# Support Vector Machine for Classification
from sklearn.svm import SVC  # Support Vector Classifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
import numpy as np

# Generate sample 2D data that's linearly separable
# WHY 2D? Easy to visualize, but concepts extend to any dimensions
X, y = make_classification(
    n_samples=200,  # 200 data points
    n_features=2,   # 2 features (x and y coordinates for visualization)
    n_informative=2,  # Both features are useful for classification
    n_redundant=0,    # No redundant/correlated features
    n_clusters_per_class=1,  # Each class forms one cluster
    random_state=42   # Reproducible results
)

# Split data into train (80%) and test (20%) sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Scale features - ABSOLUTELY CRITICAL for SVM!
# WHY? SVM uses distances between points. Features with larger ranges
# would dominate the distance calculations, making other features irrelevant
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Learn scaling from training data
X_test_scaled = scaler.transform(X_test)  # Apply same scaling to test data

print("Data shape:", X_train.shape)
print(f"Class distribution: {np.bincount(y_train)} (should be roughly balanced)\n")

# Create linear SVM classifier
# 'linear' kernel = find straight line (hyperplane) to separate classes
svm_linear = SVC(
    kernel='linear',  # Use linear kernel (straight line decision boundary)
    C=1.0,           # Regularization: lower = simpler model, higher = fit training data better
    random_state=42
)

# Train the model - this finds the optimal hyperplane
svm_linear.fit(X_train_scaled, y_train)

# Evaluate
accuracy = svm_linear.score(X_test_scaled, y_test)
print(f"Linear SVM Accuracy: {accuracy:.2%}")

# Support vectors are the critical points that define the decision boundary
print(f"Number of support vectors: {len(svm_linear.support_vectors_)}")
print(f"  → Only {len(svm_linear.support_vectors_)} out of {len(X_train)} training points actually matter!")
print(f"  → These are the points closest to the decision boundary")
print(f"  → Moving other points won't change the boundary at all")

The Kernel Trick

Real-world data often isn't linearly separable - you can't draw a straight line to separate the classes. Imagine trying to separate two groups of dots arranged in circles - no straight line works!

The kernel trick is like a magic spell that transforms your data into a higher dimension where a straight line (or hyperplane) can separate them. Think of it like this: if you have overlapping circles on a flat paper (2D), you could lift one circle up into 3D space, and now a flat plane can separate them! The amazing part is that SVM does this transformation implicitly - it never actually moves the data, but mathematically acts as if it did. This makes it computationally efficient even in very high dimensions.

# RBF Kernel - handles non-linear decision boundaries
from sklearn.datasets import make_circles

# Create non-linearly separable data
X_circles, y_circles = make_circles(
    n_samples=200, 
    noise=0.1, 
    factor=0.3,
    random_state=42
)

X_train_c, X_test_c, y_train_c, y_test_c = train_test_split(
    X_circles, y_circles, test_size=0.2, random_state=42
)

# Compare Linear vs RBF kernel
svm_linear = SVC(kernel='linear')
svm_rbf = SVC(kernel='rbf', gamma='scale')

svm_linear.fit(X_train_c, y_train_c)
svm_rbf.fit(X_train_c, y_train_c)

print(f"Linear kernel on circles: {svm_linear.score(X_test_c, y_test_c):.2%}")
print(f"RBF kernel on circles: {svm_rbf.score(X_test_c, y_test_c):.2%}")

Key Hyperparameters Explained in Detail

# Important SVM parameters - let's understand what they do
svm_tuned = SVC(
    kernel='rbf',     # Kernel choice (we'll use RBF for non-linear data)
    C=1.0,            # Regularization parameter
    gamma='scale',    # Kernel coefficient
    random_state=42
)

# ========================================
# PARAMETER 1: C (Regularization)
# ========================================
# C controls the trade-off between:
#   1. Having a smooth decision boundary (generalization)
#   2. Classifying training points correctly (accuracy on training data)

# Low C (e.g., 0.1):
#   → MORE regularization = simpler model
#   → Allows some misclassifications
#   → Wider margin (more tolerance)
#   → Better for noisy data
#   → Risk: May underfit (too simple)

# High C (e.g., 100):
#   → LESS regularization = complex model
#   → Tries to classify every training point correctly
#   → Narrow margin (less tolerance)
#   → Fits training data very closely
#   → Risk: May overfit (too specific to training data)

# Default C=1.0 is usually a good starting point

print("C Parameter Examples:")
for C_val in [0.1, 1.0, 10.0]:
    svm = SVC(kernel='rbf', C=C_val, random_state=42)
    svm.fit(X_train_scaled, y_train)
    train_acc = svm.score(X_train_scaled, y_train)
    test_acc = svm.score(X_test_scaled, y_test)
    print(f"C={C_val:5.1f}: Train={train_acc:.2%}, Test={test_acc:.2%}, "
          f"Support Vectors={len(svm.support_vectors_)}")

print("\n→ Notice: Higher C = Higher training accuracy but more support vectors")
print("→ Goal: Find C where test accuracy is highest (not training!)\n")

# ========================================
# PARAMETER 2: gamma (Kernel Coefficient)
# ========================================
# gamma controls how far the influence of a single training example reaches
# Think of it as the "reach" or "spread" of each point

# Low gamma (e.g., 0.001):
#   → Far reach = each point influences distant points
#   → Smoother decision boundaries
#   → More general, less complex
#   → Risk: May underfit

# High gamma (e.g., 10):
#   → Close reach = only nearby points are influenced
#   → Tight, wiggly decision boundaries
#   → Very specific to training data
#   → Risk: Overfits easily

# 'scale' (default): gamma = 1 / (n_features * X.var())
# 'auto': gamma = 1 / n_features

print("gamma Parameter Examples:")
for gamma_val in [0.01, 0.1, 1.0]:
    svm = SVC(kernel='rbf', gamma=gamma_val, random_state=42)
    svm.fit(X_train_scaled, y_train)
    train_acc = svm.score(X_train_scaled, y_train)
    test_acc = svm.score(X_test_scaled, y_test)
    print(f"gamma={gamma_val:5.2f}: Train={train_acc:.2%}, Test={test_acc:.2%}")

print("\n→ Notice: Higher gamma can lead to overfitting (high train, low test)")
print("→ Start with 'scale' or 'auto', then tune if needed\n")

# PRACTICAL ADVICE:
print("=== Practical Tuning Strategy ===")
print("1. Start with C=1.0 and gamma='scale'")
print("2. If underfitting: increase C (try 10, 100)")
print("3. If overfitting: decrease C (try 0.1, 0.01)")
print("4. Fine-tune gamma only after C is reasonable")
print("5. Use GridSearchCV to test combinations systematically")

# Grid search for optimal C and gamma
from sklearn.model_selection import GridSearchCV

param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': ['scale', 0.1, 0.01, 0.001],
    'kernel': ['rbf']
}

grid_search = GridSearchCV(SVC(), param_grid, cv=5, scoring='accuracy')
grid_search.fit(X_train_scaled, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV accuracy: {grid_search.best_score_:.2%}")

# STEP 1: Enable probability estimates
svm_prob = SVC(kernel='rbf', probability=True, random_state=42)
svm_prob.fit(X_train_scaled, y_train)

# STEP 2: Get probability predictions
probs = svm_prob.predict_proba(X_test_scaled)
print(f"First sample probabilities: {probs[0]}")
print(f"Shape: {probs.shape}  # (n_samples, n_classes)")

# STEP 3: Get class with highest probability
predicted_class = probs[0].argmax()
confidence = probs[0][predicted_class]
print(f"\nPredicted class: {predicted_class} with {confidence:.2%} confidence")

print("\nNote: probability=True uses 5-fold CV internally, making training slower!")

from sklearn.datasets import make_moons

# STEP 1: Create non-linear data (moon shapes)
X, y = make_moons(n_samples=200, noise=0.15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# STEP 2: Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# STEP 3: Train with linear kernel
svm_linear = SVC(kernel='linear')
svm_linear.fit(X_train_scaled, y_train)
linear_score = svm_linear.score(X_test_scaled, y_test)

# STEP 4: Train with RBF kernel
svm_rbf = SVC(kernel='rbf', gamma='scale')
svm_rbf.fit(X_train_scaled, y_train)
rbf_score = svm_rbf.score(X_test_scaled, y_test)

# STEP 5: Compare results
print(f"Linear kernel accuracy: {linear_score:.2%}")
print(f"RBF kernel accuracy: {rbf_score:.2%}")
print(f"\nRBF wins by: {rbf_score - linear_score:.2%}")
print("\nLesson: RBF kernel handles non-linear boundaries much better!")

# STEP 1: Test different C values
C_values = [0.1, 1, 10, 100]
results = []

for C in C_values:
    svm = SVC(kernel='rbf', C=C, random_state=42)
    svm.fit(X_train_scaled, y_train)
    
    train_acc = svm.score(X_train_scaled, y_train)
    test_acc = svm.score(X_test_scaled, y_test)
    gap = train_acc - test_acc
    
    results.append({
        'C': C,
        'Train': train_acc,
        'Test': test_acc,
        'Gap': gap
    })

# STEP 2: Display results
import pandas as pd
df_results = pd.DataFrame(results)
print(df_results.to_string(index=False))

# STEP 3: Interpret
print("\nObservations:")
print("- Low C (0.1): More regularization, simpler boundary, may underfit")
print("- High C (100): Less regularization, complex boundary, may overfit")
print("- Optimal C: Balance between train and test accuracy")

from sklearn.model_selection import GridSearchCV

# STEP 1: Define comprehensive parameter grid
param_grid = {
    'kernel': ['linear', 'rbf', 'poly'],
    'C': [0.1, 1, 10, 100],
    'gamma': ['scale', 'auto', 0.1, 0.01, 0.001]
}

# STEP 2: Perform grid search
grid_search = GridSearchCV(
    SVC(random_state=42),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    verbose=1
)
grid_search.fit(X_train_scaled, y_train)

# STEP 3: Display best parameters
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.2%}")
print(f"Test score: {grid_search.score(X_test_scaled, y_test):.2%}")

# STEP 4: Analyze top 5 configurations
results_df = pd.DataFrame(grid_search.cv_results_)
top_5 = results_df.nlargest(5, 'mean_test_score')[[
    'param_kernel', 'param_C', 'param_gamma', 'mean_test_score'
]]
print("\nTop 5 Configurations:")
print(top_5.to_string(index=False))

# STEP 5: Get best model
best_svm = grid_search.best_estimator_
print(f"\nNumber of support vectors: {len(best_svm.support_vectors_)}")

KNN is one of the simplest yet most intuitive classification algorithms. It classifies a new data point based on the majority vote of its K nearest neighbors - literally "tell me who your friends are, and I'll tell you who you are."

How KNN Works

KNN is called a lazy learner (or instance-based learner) because it doesn't actually "learn" anything during training - it just memorizes all the training data! Think of it like a student who doesn't study before the exam but brings all their notes and textbooks into the test.

When you ask KNN to predict a new data point, here's what happens: It looks at your new point and finds the K closest training examples (neighbors). If K=5, it finds the 5 nearest neighbors. Then it takes a democratic vote - if 4 out of 5 neighbors are "Class A", the new point is predicted as "Class A". The key idea: "You are who your friends are." Similar data points should have similar labels. This makes KNN very intuitive but can be slow on large datasets since it needs to calculate distances to all training points for every prediction.

KNN Implementation

STEP 1: Prepare Data with Scaling

# K-Nearest Neighbors Classifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
import numpy as np

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Scale features - CRITICAL for KNN!
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create KNN classifier with K=5
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)

print(f"KNN (K=5) Accuracy: {knn.score(X_test_scaled, y_test):.2%}")

Unlike other algorithms, KNN doesn't learn a model during training - it simply stores all training data. When predicting, it finds the 5 closest training samples and takes a majority vote. Scaling is critical because KNN uses distance calculations!

Choosing the Right K

The value of K is crucial and dramatically affects how your model behaves. Let's understand the trade-offs:

• Small K (e.g., 1-3): More sensitive to noise, complex decision boundaries, may overfit
  → Like asking only your closest friend for advice - if they're wrong, you're wrong. With K=1, every outlier or noisy point affects predictions. Decision boundary becomes very wiggly and complex.
• Large K (e.g., 15-25): Smoother boundaries, more robust, but may underfit
  → Like asking 20 people for advice - more stable consensus but might miss subtle patterns. Too large and it becomes like asking the entire dataset, losing the "local" aspect of nearest neighbors.
• Rule of thumb: Start with K = √n (where n is training set size)
  → For 100 samples, try K=10. This balances local and global information.
• Best practice: Use odd K for binary classification to avoid ties
  → With K=4, you might get 2 votes for each class - which one wins? Using K=5 prevents ties!

# Find optimal K using cross-validation
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt

k_range = range(1, 31)
k_scores = []

for k in k_range:
    knn_temp = KNeighborsClassifier(n_neighbors=k)
    # 5-fold cross-validation
    scores = cross_val_score(knn_temp, X_train_scaled, y_train, cv=5)
    k_scores.append(scores.mean())

# Find best K
best_k = k_range[np.argmax(k_scores)]
print(f"Best K: {best_k} with CV accuracy: {max(k_scores):.2%}")

# Plot K vs Accuracy
plt.figure(figsize=(10, 5))
plt.plot(k_range, k_scores, marker='o')
plt.xlabel('K (Number of Neighbors)')
plt.ylabel('Cross-Validation Accuracy')
plt.title('Finding Optimal K')
plt.axvline(x=best_k, color='r', linestyle='--', label=f'Best K={best_k}')
plt.legend()
plt.grid(True)
plt.show()

Distance Metrics: How KNN Measures "Closeness"

KNN relies on measuring distance between points. But what does "distance" mean? There are multiple ways to calculate it, and the choice can significantly impact your results!

# Compare different distance metrics
from sklearn.neighbors import KNeighborsClassifier

print("=== Comparing Distance Metrics ===")
metrics = ['euclidean', 'manhattan', 'chebyshev', 'minkowski']

for metric in metrics:
    # Train KNN with different distance metrics
    knn = KNeighborsClassifier(
        n_neighbors=5,
        metric=metric,
        n_jobs=-1  # Use all CPU cores
    )
    knn.fit(X_train_scaled, y_train)
    
    train_acc = knn.score(X_train_scaled, y_train)
    test_acc = knn.score(X_test_scaled, y_test)
    
    print(f"{metric:12} - Train: {train_acc:.2%}, Test: {test_acc:.2%}")

print("\n→ Try all metrics! Performance varies by dataset")
print("→ Usually Euclidean or Manhattan work best")
print("→ Cosine is special - use for text/sparse data")

Weighted KNN

# Weighted KNN - closer neighbors have more influence
knn_weighted = KNeighborsClassifier(
    n_neighbors=5,
    weights='distance'  # 'uniform' (default) or 'distance'
)
knn_weighted.fit(X_train_scaled, y_train)

print(f"Uniform weights: {knn.score(X_test_scaled, y_test):.2%}")
print(f"Distance weights: {knn_weighted.score(X_test_scaled, y_test):.2%}")

# Distance weighting: closer neighbors have more vote power
# Often helps when K is large

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

# STEP 1: Load and split data
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# STEP 2: Scale features (CRITICAL for KNN!)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# STEP 3: Train KNN with K=3
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train_scaled, y_train)

# STEP 4: Evaluate
print(f"Training Accuracy: {knn.score(X_train_scaled, y_train):.2%}")
print(f"Test Accuracy: {knn.score(X_test_scaled, y_test):.2%}")

print("\nRemember: KNN is a lazy learner - no training phase, just stores data!")

from sklearn.model_selection import cross_val_score
import numpy as np
import matplotlib.pyplot as plt

# STEP 1: Test different K values
k_range = range(1, 31)
k_scores = []

for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    # 5-fold cross-validation
    scores = cross_val_score(knn, X_train_scaled, y_train, cv=5, scoring='accuracy')
    k_scores.append(scores.mean())

# STEP 2: Find best K
best_k = k_range[np.argmax(k_scores)]
best_score = max(k_scores)
print(f"Best K: {best_k}")
print(f"Best CV Accuracy: {best_score:.2%}")

# STEP 3: Visualize K vs Accuracy
plt.figure(figsize=(10, 5))
plt.plot(k_range, k_scores, marker='o', linewidth=2)
plt.axvline(best_k, color='r', linestyle='--', label=f'Best K={best_k}')
plt.xlabel('K (Number of Neighbors)', fontsize=12)
plt.ylabel('Cross-Validation Accuracy', fontsize=12)
plt.title('Finding Optimal K for KNN', fontsize=14)
plt.grid(True, alpha=0.3)
plt.legend()
plt.show()

# STEP 4: Train with optimal K
knn_best = KNeighborsClassifier(n_neighbors=best_k)
knn_best.fit(X_train_scaled, y_train)
print(f"\nTest Accuracy with K={best_k}: {knn_best.score(X_test_scaled, y_test):.2%}")

# STEP 1: Test different distance metrics
metrics = ['euclidean', 'manhattan', 'chebyshev', 'minkowski']
results = []

for metric in metrics:
    knn = KNeighborsClassifier(n_neighbors=5, metric=metric)
    knn.fit(X_train_scaled, y_train)
    
    train_acc = knn.score(X_train_scaled, y_train)
    test_acc = knn.score(X_test_scaled, y_test)
    
    results.append({
        'Metric': metric,
        'Train Acc': f"{train_acc:.2%}",
        'Test Acc': f"{test_acc:.2%}"
    })

# STEP 2: Display results
import pandas as pd
df_results = pd.DataFrame(results)
print(df_results.to_string(index=False))

# STEP 3: Explain metrics
print("\nDistance Metrics Explained:")
print("- Euclidean: Straight-line distance (most common)")
print("- Manhattan: Sum of absolute differences (grid-like paths)")
print("- Chebyshev: Maximum difference along any dimension")
print("- Minkowski: Generalization (p=2 is Euclidean)")

# STEP 1: Train with uniform weights (default)
knn_uniform = KNeighborsClassifier(
    n_neighbors=10,  # Use larger K to see difference
    weights='uniform'  # All neighbors vote equally
)
knn_uniform.fit(X_train_scaled, y_train)

# STEP 2: Train with distance weights
knn_weighted = KNeighborsClassifier(
    n_neighbors=10,
    weights='distance'  # Closer neighbors have more influence
)
knn_weighted.fit(X_train_scaled, y_train)

# STEP 3: Get predictions with probabilities
sample = X_test_scaled[0:1]
prob_uniform = knn_uniform.predict_proba(sample)[0]
prob_weighted = knn_weighted.predict_proba(sample)[0]

print("=== Sample Prediction Probabilities ===")
print(f"Uniform weights:   {prob_uniform}")
print(f"Distance weights:  {prob_weighted}")

# STEP 4: Compare accuracies
print("\n=== Accuracy Comparison ===")
print(f"Uniform KNN:   {knn_uniform.score(X_test_scaled, y_test):.2%}")
print(f"Weighted KNN:  {knn_weighted.score(X_test_scaled, y_test):.2%}")

# STEP 5: Test on different K values
print("\n=== K Value Impact ===")
for k in [3, 5, 10, 15, 20]:
    knn_u = KNeighborsClassifier(n_neighbors=k, weights='uniform')
    knn_w = KNeighborsClassifier(n_neighbors=k, weights='distance')
    
    knn_u.fit(X_train_scaled, y_train)
    knn_w.fit(X_train_scaled, y_train)
    
    acc_u = knn_u.score(X_test_scaled, y_test)
    acc_w = knn_w.score(X_test_scaled, y_test)
    
    print(f"K={k:2d}  Uniform: {acc_u:.2%}  Weighted: {acc_w:.2%}  Diff: {acc_w-acc_u:+.2%}")

print("\nInsight: Distance weighting often helps with larger K values!")

Naive Bayes is a fast, probabilistic classifier based on Bayes' theorem with a "naive" assumption of feature independence. Despite its simplicity, it often performs surprisingly well, especially for text classification tasks like spam detection.

Bayes' Theorem

Naive Bayes applies Bayes' theorem to calculate probabilities. Let's understand it with an example: If you see an email with words "free", "winner", and "claim", what's the probability it's spam?

Bayes' theorem lets us flip the question around. Instead of asking "What's P(spam | these words)?", we calculate it from easier-to-find probabilities: "What's P(these words | spam)?" and "What's P(spam in general)?" This clever reversal makes the math much simpler!

Types of Naive Bayes: Which One to Use?

There are three main types of Naive Bayes classifiers, each designed for different kinds of data. Choosing the right one is crucial for good performance!

Gaussian Naive Bayes

STEP 1: Train Gaussian NB on Continuous Features

# Gaussian Naive Bayes for continuous features
from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# Load Iris data (continuous features)
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train Gaussian Naive Bayes
gnb = GaussianNB()
gnb.fit(X_train, y_train)

# Predictions
y_pred = gnb.predict(X_test)

print(f"Gaussian NB Accuracy: {accuracy_score(y_test, y_pred):.2%}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))

Gaussian NB assumes each feature follows a normal distribution within each class. It calculates mean and standard deviation for each feature per class during training, then uses Bayes' theorem to compute class probabilities for new samples.

STEP 2: Examine Probability Outputs

# Examine class probabilities
sample = X_test[0:1]
proba = gnb.predict_proba(sample)
print(f"Sample features: {sample[0]}")
print(f"Class probabilities: {proba[0]}")
print(f"Predicted class: {iris.target_names[gnb.predict(sample)[0]]}")

Multinomial Naive Bayes for Text

Multinomial NB is the go-to algorithm for text classification. It works with word counts or TF-IDF features.

STEP 1: Convert Text to Features and Train

# Text Classification with Multinomial Naive Bayes
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

# Sample text data
texts = [
    "Free money!!! Click here to win big prizes",
    "Urgent: You have won a lottery!",
    "Meeting tomorrow at 10am in conference room",
    "Project deadline extended to next Friday",
    "Congratulations! You've been selected for a prize",
    "Can we schedule a call for next week?",
    "FREE iPhone!!! Act now limited time offer",
    "Quarterly report is ready for review"
]
labels = [1, 1, 0, 0, 1, 0, 1, 0]  # 1 = spam, 0 = not spam

# Convert text to features using TF-IDF
vectorizer = TfidfVectorizer()
X_text = vectorizer.fit_transform(texts)

# Train Multinomial Naive Bayes
mnb = MultinomialNB()
mnb.fit(X_text, labels)

# Test on new messages
new_messages = [
    "Win a free vacation today!",
    "Meeting rescheduled to 3pm"
]
new_vectors = vectorizer.transform(new_messages)
predictions = mnb.predict(new_vectors)

for msg, pred in zip(new_messages, predictions):
    print(f"'{msg}' → {'SPAM' if pred else 'NOT SPAM'}")

The vectorizer converts text to numerical features (word importance scores). Multinomial NB then calculates P(Spam|words) vs P(Not Spam|words) using Bayes' theorem. Words like "free", "win", "urgent" increase spam probability.

Bernoulli Naive Bayes

# Bernoulli NB for binary features
from sklearn.naive_bayes import BernoulliNB

# Binary features (word present = 1, absent = 0)
vectorizer_binary = CountVectorizer(binary=True)
X_binary = vectorizer_binary.fit_transform(texts)

# Train Bernoulli NB
bnb = BernoulliNB()
bnb.fit(X_binary, labels)

# Bernoulli NB explicitly models absence of features
# Good when "word NOT present" is informative

Advantages & Use Cases

# Smoothing parameter (Laplace smoothing)
# Prevents zero probability for unseen feature values
mnb_smoothed = MultinomialNB(alpha=1.0)  # Default Laplace smoothing
mnb_no_smooth = MultinomialNB(alpha=0.0)  # No smoothing (risky!)

# alpha = 1.0: Laplace smoothing (adds 1 to all counts)
# alpha = 0.5: Lidstone smoothing
# alpha = 0.0: No smoothing (not recommended)

print("Smoothing prevents zero probabilities from unseen words!")

from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer

# STEP 1: Create training data
texts = [
    "Free money!!! Click here to win",
    "Urgent: You won a lottery!",
    "Meeting at 10am in conference room",
    "Project deadline extended",
    "Congratulations! You've been selected",
    "Can we schedule a call?",
    "FREE iPhone!!! Act now",
    "Quarterly report ready"
]
labels = [1, 1, 0, 0, 1, 0, 1, 0]  # 1=spam, 0=ham

# STEP 2: Create pipeline
spam_classifier = Pipeline([
    ('vectorizer', TfidfVectorizer(
        stop_words='english',  # Remove common words
        max_features=100       # Limit vocabulary
    )),
    ('classifier', MultinomialNB())
])

# STEP 3: Train (one command!)
spam_classifier.fit(texts, labels)

# STEP 4: Test on new messages
test_msgs = [
    "Buy now! Limited offer!",
    "See you at lunch tomorrow",
    "WINNER! Claim your prize now!",
    "Can you review the document?"
]

predictions = spam_classifier.predict(test_msgs)
probs = spam_classifier.predict_proba(test_msgs)

for msg, pred, prob in zip(test_msgs, predictions, probs):
    label = 'SPAM' if pred else 'HAM'
    confidence = prob[pred]
    print(f"{label} ({confidence:.2%}): {msg}")

from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score

# STEP 1: Load Iris data
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# STEP 2: Train Gaussian NB (for continuous features)
gnb = GaussianNB()
gnb.fit(X_train, y_train)
gnb_score = gnb.score(X_test, y_test)
gnb_cv = cross_val_score(gnb, X, y, cv=5).mean()

# STEP 3: Train Multinomial NB (requires non-negative features)
# Iris has some small values, so we'll shift to make all positive
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
X_scaled = scaler.fit_transform(X)

mnb = MultinomialNB()
mnb.fit(X_train_scaled, y_train)
mnb_score = mnb.score(X_test_scaled, y_test)
mnb_cv = cross_val_score(mnb, X_scaled, y, cv=5).mean()

# STEP 4: Compare results
print("=== Performance Comparison ===")
print(f"Gaussian NB:    Test={gnb_score:.2%}  CV={gnb_cv:.2%}")
print(f"Multinomial NB: Test={mnb_score:.2%}  CV={mnb_cv:.2%}")
print("\nGaussian NB is better for continuous features like Iris!")

from sklearn.model_selection import GridSearchCV
import numpy as np

# STEP 1: Prepare text data (using previous spam example)
vectorizer = TfidfVectorizer(stop_words='english')
X_text = vectorizer.fit_transform(texts)

# STEP 2: Test different alpha values
alphas = [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0]
results = []

for alpha in alphas:
    mnb = MultinomialNB(alpha=alpha)
    scores = cross_val_score(mnb, X_text, labels, cv=3)
    results.append({
        'Alpha': alpha,
        'CV Score': scores.mean(),
        'Std': scores.std()
    })

# STEP 3: Display results
import pandas as pd
df_results = pd.DataFrame(results)
print(df_results.to_string(index=False))

# STEP 4: Use GridSearchCV for optimal alpha
param_grid = {'alpha': np.logspace(-3, 1, 20)}
grid = GridSearchCV(MultinomialNB(), param_grid, cv=3, scoring='accuracy')
grid.fit(X_text, labels)

print(f"\nBest alpha: {grid.best_params_['alpha']:.4f}")
print(f"Best CV score: {grid.best_score_:.2%}")

# STEP 5: Explain smoothing
print("\nSmoothing (alpha) explained:")
print("  alpha=0: No smoothing (risky - zero probabilities possible)")
print("  alpha=1: Laplace smoothing (default, adds 1 to all counts)")
print("  alpha>1: More smoothing (for very sparse data)")

# STEP 1: Create sentiment dataset
reviews = [
    "This movie was absolutely amazing! Loved it!",
    "Terrible film, waste of time and money",
    "Brilliant acting and great storyline",
    "Boring and predictable, very disappointed",
    "Outstanding performance, highly recommend",
    "Awful movie, couldn't wait for it to end",
    "Fantastic cinematography and soundtrack",
    "Poor script and bad acting",
    "Incredible experience, must watch!",
    "Worst movie I've ever seen"
]
sentiments = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0]  # 1=positive, 0=negative

# STEP 2: Train sentiment classifier
vectorizer = TfidfVectorizer(stop_words='english', max_features=50)
X_reviews = vectorizer.fit_transform(reviews)

mnb = MultinomialNB()
mnb.fit(X_reviews, sentiments)

# STEP 3: Extract feature importances
feature_names = vectorizer.get_feature_names_out()
log_prob_pos = mnb.feature_log_prob_[1]  # Positive class
log_prob_neg = mnb.feature_log_prob_[0]  # Negative class

# Words most indicative of positive sentiment
pos_words = pd.DataFrame({
    'Word': feature_names,
    'Positive Score': log_prob_pos
}).nlargest(10, 'Positive Score')

# Words most indicative of negative sentiment
neg_words = pd.DataFrame({
    'Word': feature_names,
    'Negative Score': log_prob_neg
}).nlargest(10, 'Negative Score')

print("=== Top Positive Words ===")
print(pos_words.to_string(index=False))
print("\n=== Top Negative Words ===")
print(neg_words.to_string(index=False))

# STEP 4: Test on new reviews
test_reviews = [
    "Amazing movie, absolutely brilliant!",
    "Terrible and boring, don't waste your time"
]

test_vectors = vectorizer.transform(test_reviews)
predictions = mnb.predict(test_vectors)
probs = mnb.predict_proba(test_vectors)

print("\n=== Sentiment Predictions ===")
for review, pred, prob in zip(test_reviews, predictions, probs):
    sentiment = 'POSITIVE' if pred else 'NEGATIVE'
    confidence = prob[pred]
    print(f"{sentiment} ({confidence:.2%}): {review}")

print("\nThis demonstrates how Naive Bayes learns word-sentiment associations!")