Logistic Regression

So far, we have learned about Linear Regression, which predicts continuous values like house prices or temperature. But many real-world problems require different answers. Should we approve this loan? Will this patient develop diabetes? Is this email spam? These are classification problems where we predict categories, not numbers. Logistic Regression is the go-to algorithm for binary classification - predicting one of two possible outcomes.

Regression vs Classification

Let's clarify the difference between the two main types of supervised learning. Regression predicts continuous numerical outputs from a range of infinite possibilities. Classification predicts discrete categorical outputs from a finite set of classes. Despite its name, Logistic Regression is actually a classification algorithm, not a regression algorithm.

# Understanding the difference: Regression vs Classification

# REGRESSION: Predicting continuous values
# Example: House price prediction
house_features = {"bedrooms": 3, "sqft": 1500, "location": "suburban"}
predicted_price = 285000.50  # Could be any number
print(f"Predicted house price: ${predicted_price:,.2f}")

# CLASSIFICATION: Predicting discrete categories
# Example: Email spam detection
email_features = {"has_link": True, "urgent_words": 5, "sender": "unknown"}
predicted_class = "spam"  # Only two options: "spam" or "not spam"
print(f"Email classification: {predicted_class}")

# Key difference:
print("\nRegression output: Any number (infinite possibilities)")
print("Classification output: One of fixed categories (finite choices)")

Binary vs Multiclass Classification

Classification problems come in two main flavors. Binary classification has two possible outcomes (spam or not spam, disease or healthy). Multiclass classification has three or more categories (is this email spam, promotional, or personal?). Logistic Regression handles binary classification directly, though we can extend it to multiclass problems using techniques like One-vs-Rest.

# Binary vs Multiclass Classification Examples

# BINARY CLASSIFICATION (2 classes)
# Logistic Regression is perfect for this!
binary_examples = {
    "Email": ["spam", "not spam"],
    "Medical": ["disease", "healthy"],
    "Loan": ["approved", "rejected"],
    "Fraud": ["fraudulent", "legitimate"]
}

print("Binary Classification Examples:")
for task, classes in binary_examples.items():
    print(f"  {task}: {classes[0]} vs {classes[1]}")

# MULTICLASS CLASSIFICATION (3+ classes)
# Requires extension like One-vs-Rest
multiclass_examples = {
    "Digit Recognition": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
    "Email Type": ["spam", "promotional", "personal", "work"],
    "Sentiment": ["negative", "neutral", "positive"]
}

print("\nMulticlass Classification Examples:")
for task, classes in multiclass_examples.items():
    print(f"  {task}: {len(classes)} classes")

Why Linear Regression Won't Work

You might ask: why can't we just use Linear Regression and round the output to 0 or 1? Let's see what happens. Recall that Linear Regression creates a line (or hyperplane in multiple dimensions) that fits the data. For classification, we want probabilities between 0 and 1, but Linear Regression can predict values far outside this range.

# Linear Regression on classification data
from sklearn.linear_model import LinearRegression
import numpy as np

# Example: predicting if student passes (1) or fails (0) based on hours studied
X = np.array([[2], [3], [4], [5], [6], [7], [8], [9]])
y = np.array([0, 0, 0, 1, 1, 1, 1, 1])

# Fit Linear Regression
model = LinearRegression()
model.fit(X, y)

# Predictions
predictions = model.predict(np.array([[0], [10], [15]]))
print("Hour 0:", predictions[0])  # -0.34 (negative probability!)
print("Hour 10:", predictions[1])  # 1.19 (probability > 1!)
print("Hour 15:", predictions[2])  # 2.09 (nonsensical!)

As you can see, Linear Regression produces predictions outside the valid probability range [0, 1]. This is why we need a special algorithm for classification: Logistic Regression. It uses the sigmoid function to squash any input value into the range 0 to 1.

problems = [
    "Predicting house prices",
    "Identifying if a photo contains a cat",
    "Forecasting next month's revenue",
    "Detecting credit card fraud"
]

problem_types = {
    "Predicting house prices": "regression",
    "Identifying if a photo contains a cat": "classification",
    "Forecasting next month's revenue": "regression",
    "Detecting credit card fraud": "classification"
}

for problem, ptype in problem_types.items():
    print(f"{problem}: {ptype}")

from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[2], [3], [4], [5], [6], [7], [8], [9]])
y = np.array([0, 0, 0, 1, 1, 1, 1, 1])

from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[2], [3], [4], [5], [6], [7], [8], [9]])
y = np.array([0, 0, 0, 1, 1, 1, 1, 1])

model = LinearRegression()
model.fit(X, y)

pred_0 = model.predict([[0]])[0]
pred_12 = model.predict([[12]])[0]

print(f"Prediction at X=0: {pred_0:.3f}")   # Negative!
print(f"Prediction at X=12: {pred_12:.3f}") # Greater than 1!
print("\nThese are invalid probabilities!")

import numpy as np
X = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]).reshape(-1, 1)
y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])

import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression

X = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]).reshape(-1, 1)
y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])

linear_model = LinearRegression()
logistic_model = LogisticRegression()

linear_model.fit(X, y)
logistic_model.fit(X, y)

test_values = [[-2], [5], [12]]

print("X value | Linear | Logistic Prob")
print("-" * 35)
for x in test_values:
    lin_pred = linear_model.predict([x])[0]
    log_prob = logistic_model.predict_proba([x])[0, 1]
    print(f"{x[0]:7} | {lin_pred:6.3f} | {log_prob:.3f}")

The heart of Logistic Regression is the sigmoid function, an elegant mathematical tool that transforms any input value into a probability between 0 and 1. This S-shaped curve is perfect for classification because it maps negative infinity to 0, zero to 0.5, and positive infinity to 1. The sigmoid function is the key innovation that makes Logistic Regression work.

The Sigmoid Function Equation

The sigmoid function, also called the logistic function, is mathematically expressed as:

This simple equation is incredibly powerful. Let's understand what happens at key points:

• z = -5: σ(z) ≈ 0.0067 (very close to 0)
• z = 0: σ(z) = 0.5 (exactly halfway)
• z = 5: σ(z) ≈ 0.9933 (very close to 1)

Let's implement this in Python step by step:

Step 1: Define the Sigmoid Function

First, we create a simple function that takes any number z and returns the sigmoid value. We use Python's math.exp() for the exponential calculation.

# Import the math module for exponential function
import math

def sigmoid(z):
    """Calculate sigmoid of z"""
    return 1 / (1 + math.exp(-z))

Step 2: Test with Negative Input (z = -5)

When z is negative, the sigmoid outputs a value close to 0. This represents low probability of belonging to class 1.

# When z = -5 (negative input)
z = -5
exp_term = math.exp(-z)  # e^5 ≈ 148.41
result = 1 / (1 + exp_term)

print(f"z = {z}: sigmoid = {result:.4f}")
# Output: z = -5: sigmoid = 0.0067

Step 3: Test with Zero Input (z = 0)

When z = 0, the sigmoid returns exactly 0.5. This is the decision boundary where the model is equally uncertain about both classes.

# When z = 0 (neutral input)
z = 0
exp_term = math.exp(-z)  # e^0 = 1
result = 1 / (1 + exp_term)

print(f"z = {z}: sigmoid = {result:.4f}")
# Output: z = 0: sigmoid = 0.5000

Step 4: Test with Positive Input (z = 5)

When z is positive, the sigmoid outputs a value close to 1. This represents high probability of belonging to class 1.

# When z = 5 (positive input)
z = 5
exp_term = math.exp(-z)  # e^-5 ≈ 0.0067
result = 1 / (1 + exp_term)

print(f"z = {z}: sigmoid = {result:.4f}")
# Output: z = 5: sigmoid = 0.9933

The sigmoid function has a beautiful S-shaped curve. Small changes in input near z=0 cause large changes in output, while extreme values flatten out. This makes the sigmoid ideal for representing probabilities.

Visualizing the Sigmoid Function

A picture is worth a thousand words. Let's visualize the sigmoid curve to understand its behavior better.

Setting Up the Visualization

We'll use NumPy for efficient calculations and Matplotlib for plotting. NumPy's version of sigmoid can handle arrays of values at once.

# Import visualization libraries
import numpy as np
import matplotlib.pyplot as plt

# Define sigmoid for arrays (NumPy version)
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

Generate Input Values

We create 100 evenly spaced points between -5 and 5 to get a smooth curve.

# Generate input values from -5 to 5
z = np.linspace(-5, 5, 100)

# Calculate sigmoid for all values at once
sigma_z = sigmoid(z)

print(f"Input range: {z.min():.1f} to {z.max():.1f}")
print(f"Output range: {sigma_z.min():.4f} to {sigma_z.max():.4f}")

Create the Plot

Now we plot the S-curve with helpful reference lines at the decision boundary (0.5) and at z=0.

# Create the visualization
plt.figure(figsize=(10, 6))
plt.plot(z, sigma_z, 'b-', linewidth=2, label='σ(z)')

# Add reference lines
plt.axhline(y=0.5, color='r', linestyle='--', alpha=0.5, label='Threshold')
plt.axvline(x=0, color='gray', linestyle='-', alpha=0.3)

# Customize appearance
plt.xlabel('z (input)', fontsize=12)
plt.ylabel('σ(z) (probability)', fontsize=12)
plt.title('The Sigmoid Function', fontsize=14)
plt.legend()
plt.grid(True, alpha=0.3)
plt.ylim(-0.1, 1.1)
plt.show()

From Linear to Logistic Model

In Linear Regression, we computed a weighted sum of input features:

In Logistic Regression, we wrap this linear combination in the sigmoid function:

The sigmoid function ensures that the output is always a valid probability between 0 and 1, regardless of how large or small the linear combination becomes.

Let's see how this works with a real example. Imagine we're predicting whether a student will pass an exam based on two features: hours studied and previous test score.

Step 1: Set Up the Model

First, we import NumPy and define our sigmoid function. Then we set up the model weights. In a real scenario, these weights would be learned from training data, but here we'll use pre-defined values.

# Import library and define sigmoid
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

Step 2: Define Model Weights

The model has three weights: an intercept (bias) and one weight for each feature. These weights determine how much each feature influences the prediction.

# Model weights (learned from training data)
w0 = -4.0   # Intercept (bias)
w1 = 0.5    # Weight for hours studied
w2 = 0.3    # Weight for previous score

print("Model Weights:")
print(f"  w0 (intercept): {w0}")
print(f"  w1 (hours studied): {w1}")
print(f"  w2 (previous score): {w2}")

Step 3: Predict for Student 1 (Good Student)

Let's predict whether a student who studied 8 hours and had a previous score of 70 will pass.

# Student 1: studied 8 hours, previous score 70
hours_studied = 8
previous_score = 70

# Step A: Calculate linear combination
z = w0 + w1*hours_studied + w2*previous_score
print(f"Linear combination: {w0} + {w1}*{hours_studied} + {w2}*{previous_score}")
print(f"z = {z}")

# Step B: Apply sigmoid
probability = sigmoid(z)
print(f"Probability = sigmoid({z}) = {probability:.4f}")

# Step C: Make prediction
prediction = "PASS" if probability >= 0.5 else "FAIL"
print(f"Prediction: {prediction} ({probability*100:.1f}% confidence)")

Step 4: Predict for Student 2 (Struggling Student)

Now let's predict for a student who only studied 2 hours and had a previous score of 40.

# Student 2: studied 2 hours, previous score 40
hours_studied = 2
previous_score = 40

# Calculate linear combination
z = w0 + w1*hours_studied + w2*previous_score
print(f"Linear combination: {w0} + {w1}*{hours_studied} + {w2}*{previous_score}")
print(f"z = {z}")

# Apply sigmoid
probability = sigmoid(z)
print(f"Probability = sigmoid({z}) = {probability:.4f}")

# Make prediction
prediction = "PASS" if probability >= 0.5 else "FAIL"
print(f"Prediction: {prediction} ({probability*100:.1f}% confidence)")

Understanding the Sigmoid Mathematically

Let's break down the sigmoid function to understand why it has its special S-shape. The key is the exponential term $e^{-z}$. We'll visualize each component step by step.

Step 1: Set Up the Functions

Before we can visualize anything, we need to import our tools and define our functions. Think of this as gathering your ingredients before cooking.

# Import libraries
import numpy as np
import matplotlib.pyplot as plt

Now let's define our two functions. We'll create them separately so we can examine each piece:

# Define sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Define just the exponential component
def exp_term(z):
    return np.exp(-z)

Step 2: Generate Input Values

To draw a smooth curve, we need many points. Instead of typing each number manually, we use NumPy's linspace function to generate them automatically.

# Generate 100 points from -5 to 5
z = np.linspace(-5, 5, 100)

print(f"First 5 values: {z[:5]}")
print(f"Last 5 values: {z[-5:]}")

Step 3: Visualize the Exponential Component

The exponential term $e^{-z}$ is the secret ingredient that gives sigmoid its special properties. Let's see what it looks like when we plot it.

# Plot the exponential component
plt.figure(figsize=(8, 5))
plt.plot(z, exp_term(z), 'r-', linewidth=2)
plt.title('Component: exp(-z)', fontsize=14)
plt.xlabel('z', fontsize=12)
plt.ylabel('exp(-z)', fontsize=12)
plt.grid(True, alpha=0.3)
plt.ylim(0, 150)
plt.show()

Step 4: Visualize the Denominator

Now we add 1 to the exponential term. This might seem like a small change, but it's actually very important! This sum becomes the denominator of our sigmoid formula: $1 + e^{-z}$.

# Plot the denominator
plt.figure(figsize=(8, 5))
plt.plot(z, 1 + exp_term(z), 'g-', linewidth=2)
plt.title('Component: 1 + exp(-z)', fontsize=14)
plt.xlabel('z', fontsize=12)
plt.ylabel('1 + exp(-z)', fontsize=12)
plt.grid(True, alpha=0.3)
plt.show()

Step 5: The Final Sigmoid Curve

Now for the grand finale! We divide 1 by the denominator to get the sigmoid. This creates the famous S-shaped curve that makes logistic regression work.

# Plot the final sigmoid
plt.figure(figsize=(8, 5))
plt.plot(z, sigmoid(z), 'b-', linewidth=2)
plt.axhline(y=0.5, color='r', linestyle='--', alpha=0.5)
plt.axvline(x=0, color='gray', linestyle='-', alpha=0.3)
plt.title('Result: σ(z) = 1 / (1 + exp(-z))', fontsize=14)
plt.xlabel('z', fontsize=12)
plt.ylabel('σ(z)', fontsize=12)
plt.grid(True, alpha=0.3)
plt.ylim(-0.1, 1.1)
plt.show()

Step 6: See the Transformation in Action

Let's see exactly how different input values get transformed into probabilities. This will help you build intuition about what the sigmoid is doing.

# Show the transformation
print("Linear output -> Sigmoid output (probability)")
print("-" * 40)

for linear_output in [-10, -5, 0, 5, 10]:
    prob = sigmoid(linear_output)
    print(f"  z = {linear_output:3} -> σ(z) = {prob:.6f}")

Here are the results in a nice table:

import math

# σ(0) = 1 / (1 + e^-0)
#      = 1 / (1 + e^0)
#      = 1 / (1 + 1)
#      = 1 / 2
#      = 0.5

z = 0
sigma = 1 / (1 + math.exp(-z))
print(f"σ(0) = {sigma}")  # 0.5

test_values = [-5, -2, 0, 2, 5]

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

test_values = [-5, -2, 0, 2, 5]

print("Testing σ(-z) = 1 - σ(z):")
for z in test_values:
    sig_z = sigmoid(z)
    sig_neg_z = sigmoid(-z)
    one_minus = 1 - sig_z
    match = np.isclose(sig_neg_z, one_minus)
    print(f"z={z:2}: σ(-z)={sig_neg_z:.4f}, 1-σ(z)={one_minus:.4f}, Match: {match}")

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_derivative(z):
    sig = sigmoid(z)
    return sig * (1 - sig)

z = np.linspace(-5, 5, 100)

plt.figure(figsize=(10, 5))
plt.plot(z, sigmoid(z), 'b-', label='σ(z)')
plt.plot(z, sigmoid_derivative(z), 'r-', label="σ'(z)")
plt.axhline(y=0.25, color='g', linestyle='--', alpha=0.5)
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

print(f"Maximum derivative: σ'(0) = {sigmoid_derivative(0):.4f}")

While Logistic Regression outputs probabilities, we often need to make a definitive class prediction: yes or no, spam or not spam, disease or healthy. This is where the decision boundary comes in. The decision boundary is a threshold that separates the two classes in our classification problem. Understanding how to set and interpret this boundary is crucial for building effective classifiers.

The 0.5 Probability Threshold

# Understanding the 0.5 Threshold: Step by Step
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Simulate predictions for 10 samples
np.random.seed(42)
probabilities = np.random.uniform(0, 1, 10)

print("Decision Boundary at 0.5:")
print("="*50)
print(f"{'Sample':<10}{'Probability':<15}{'Prediction':<12}{'Reason'}")
print("-"*50)

for i, prob in enumerate(probabilities):
    if prob >= 0.5:
        prediction = "Class 1"
        reason = f"{prob:.3f} >= 0.5"
    else:
        prediction = "Class 0"
        reason = f"{prob:.3f} < 0.5"
    
    print(f"{i+1:<10}{prob:<15.4f}{prediction:<12}{reason}")

# Summary
class_1_count = sum(1 for p in probabilities if p >= 0.5)
class_0_count = len(probabilities) - class_1_count
print("-"*50)
print(f"Total Class 0: {class_0_count}, Class 1: {class_1_count}")

Geometric Interpretation

In higher dimensions, the decision boundary is a line or hyperplane that separates the feature space into regions for each class. Let's visualize this with a 2D example.

Step 1: Define Model Parameters

import numpy as np

# Trained model parameters (example)
w0 = -3.0    # Intercept (bias)
w1 = 1.5     # Weight for feature x1
w2 = 2.0     # Weight for feature x2

Step 2: Understand the Linear Combination

print("Linear combination: z = w0 + w1*x1 + w2*x2")
print(f"z = {w0} + {w1}*x1 + {w2}*x2")

# Output: z = -3.0 + 1.5*x1 + 2.0*x2

Step 3: Find Where the Decision Boundary Occurs

# Decision boundary occurs where sigmoid(z) = 0.5
# This happens when z = 0
print("Decision boundary is where z = 0 (sigmoid = 0.5)")
print(f"{w0} + {w1}*x1 + {w2}*x2 = 0")

# Output: -3.0 + 1.5*x1 + 2.0*x2 = 0

Step 4: Solve for the Boundary Line Equation

# Solve for x2 in terms of x1
print("Solving for x2:")
print(f"x2 = ({-w0} - {w1}*x1) / {w2}")
print(f"x2 = {-w0/w2:.2f} - {w1/w2:.2f}*x1")

# Output: x2 = 1.50 - 0.75*x1

Step 5: Verify with Actual Points

# Verify with some points on the boundary
print("Points on the decision boundary:")
for x1 in [0, 1, 2, 3]:
    x2 = (-w0 - w1*x1) / w2
    z = w0 + w1*x1 + w2*x2
    print(f"  x1={x1}: x2={x2:.2f} -> z={z:.4f}")

# Output:
#   x1=0: x2=1.50 -> z=0.0000
#   x1=1: x2=0.75 -> z=0.0000
#   x1=2: x2=0.00 -> z=0.0000
#   x1=3: x2=-0.75 -> z=0.0000

Step 6: Visualize the Decision Boundary

import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_blobs

# Create synthetic 2D dataset
np.random.seed(42)
X, y = make_blobs(n_samples=100, n_features=2, 
                  centers=2, random_state=42)

# Train model
model = LogisticRegression()
model.fit(X, y)

Step 7: Create a Mesh Grid for Visualization

# Create mesh grid covering the feature space
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100),
                      np.linspace(y_min, y_max, 100))

# Get probability predictions for every point
Z = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
Z = Z.reshape(xx.shape)

Step 8: Plot Everything Together

# Create the plot
fig, ax = plt.subplots(figsize=(10, 8))

# Color regions by predicted class
ax.contourf(xx, yy, Z, levels=[0, 0.5, 1], 
           colors=['lightblue', 'lightcoral'], alpha=0.6)

# Draw decision boundary (0.5 probability line)
ax.contour(xx, yy, Z, levels=[0.5], 
          colors='black', linewidths=2)

# Add probability contour lines
contours = ax.contour(xx, yy, Z, 
                     levels=[0.1, 0.3, 0.7, 0.9], 
                     colors='gray', alpha=0.4)
ax.clabel(contours, inline=True, fontsize=8)

# Plot actual data points
ax.scatter(X[y==0, 0], X[y==0, 1], c='blue', 
          marker='o', s=100, label='Class 0')
ax.scatter(X[y==1, 0], X[y==1, 1], c='red', 
          marker='s', s=100, label='Class 1')

ax.set_xlabel('Feature 1', fontsize=12)
ax.set_ylabel('Feature 2', fontsize=12)
ax.set_title('Decision Boundary with Probability Contours', 
            fontsize=14)
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Adjusting the Threshold

Different problems require different thresholds. A medical diagnostic test might need a lower threshold (more positive predictions, fewer false negatives) to catch all potential cases. A spam filter might prefer a higher threshold (fewer false positives) to avoid marking legitimate emails as spam.

Step 1: Setup the Scenario

import numpy as np

# Model probabilities for 10 patients (disease screening)
patient_probs = [0.15, 0.25, 0.35, 0.45, 0.55, 
                 0.65, 0.75, 0.85, 0.90, 0.95]

# Ground truth: who actually has the disease
actual_disease = [0, 0, 1, 0, 1, 1, 1, 1, 1, 1]

Step 2: Test Different Thresholds

print("Impact of Different Thresholds:")
print("="*60)

for threshold in [0.3, 0.5, 0.7]:
    # Apply threshold to convert probabilities to predictions
    predicted = [1 if p >= threshold else 0 
                 for p in patient_probs]
    
    print(f"\nThreshold: {threshold}")
    print(f"  Patients diagnosed: {sum(predicted)}")

Step 3: Calculate Performance Metrics

    # Count different types of outcomes
    tp = sum(1 for p, a in zip(predicted, actual_disease) 
             if p == 1 and a == 1)  # Correctly found disease
    
    fp = sum(1 for p, a in zip(predicted, actual_disease) 
             if p == 1 and a == 0)  # False alarm (healthy diagnosed as sick)
    
    fn = sum(1 for p, a in zip(predicted, actual_disease) 
             if p == 0 and a == 1)  # Missed case (sick patient missed)
    
    print(f"  True Positives: {tp} (correctly found disease)")
    print(f"  False Positives: {fp} (unnecessary worry)")
    print(f"  Missed Cases: {fn} (dangerous!)")

Step 4: Interpret the Results

    if threshold == 0.3:
        print("  -> Low threshold: Catches more disease")
        print("     but more false alarms")
    elif threshold == 0.7:
        print("  -> High threshold: Fewer false alarms")
        print("     but may miss disease!")

# Example output:
# Threshold: 0.3 -> Diagnoses 8 patients, TP=7, FP=1, FN=0
# Threshold: 0.5 -> Diagnoses 6 patients, TP=6, FP=0, FN=1
# Threshold: 0.7 -> Diagnoses 4 patients, TP=4, FP=0, FN=3

Step 5: Real Dataset Example - Load and Prepare

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Load breast cancer dataset
data = load_breast_cancer()
X, y = data.data[:, :2], data.target

# Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

Step 6: Train Model and Get Probabilities

# Train logistic regression model
model = LogisticRegression()
model.fit(X_train, y_train)

# Get probability predictions for test set
y_prob = model.predict_proba(X_test)[:, 1]

print(f"Model trained on {len(X_train)} samples")
print(f"Testing on {len(X_test)} samples")
print(f"Probability range: {y_prob.min():.3f} to {y_prob.max():.3f}")

Step 7: Compare Threshold Performance

from sklearn.metrics import confusion_matrix

thresholds = [0.3, 0.5, 0.7]
print("Threshold Performance Comparison:")
print("-" * 60)

for threshold in thresholds:
    # Apply custom threshold
    y_pred = (y_prob >= threshold).astype(int)
    
    # Get confusion matrix
    tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
    
    # Calculate metrics
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    
    print(f"\nThreshold: {threshold}")
    print(f"  TP={tp}, FP={fp}, FN={fn}, TN={tn}")
    print(f"  Precision: {precision:.3f}")
    print(f"  Recall: {recall:.3f}")

Class Probability and Confidence

The closer a probability is to 0 or 1, the more confident the model is in its prediction. Probabilities near 0.5 indicate uncertainty. This information can be valuable for decision-making.

# Understanding Prediction Confidence
import numpy as np
from sklearn.linear_model import LogisticRegression

# Sample predictions with different confidence levels
probabilities = np.array([0.05, 0.15, 0.45, 0.51, 0.85, 0.95])
threshold = 0.5

print("Prediction Confidence Analysis:")
print("-" * 50)
for prob in probabilities:
    prediction = 1 if prob >= threshold else 0
    confidence = abs(prob - 0.5) * 2  # 0-1 scale
    uncertainty = 1 - confidence
    
    print(f"Probability: {prob:.2f} -> Predict: {prediction}, Confidence: {confidence:.1%}")

probabilities = [0.2, 0.5, 0.7, 0.49]
threshold = 0.5

probabilities = [0.2, 0.5, 0.7, 0.49]
threshold = 0.5

predictions = [1 if p >= threshold else 0 for p in probabilities]
print(f"Probabilities: {probabilities}")
print(f"Predictions:   {predictions}")  # [0, 1, 1, 0]

import numpy as np
from sklearn.metrics import confusion_matrix

y_prob = np.array([0.1, 0.3, 0.4, 0.6, 0.7, 0.9])
y_true = np.array([0, 0, 0, 1, 1, 1])

import numpy as np
from sklearn.metrics import confusion_matrix

y_prob = np.array([0.1, 0.3, 0.4, 0.6, 0.7, 0.9])
y_true = np.array([0, 0, 0, 1, 1, 1])

for threshold in [0.3, 0.5, 0.7]:
    y_pred = (y_prob >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    accuracy = (tp + tn) / len(y_true)
    print(f"Threshold {threshold}: Acc={accuracy:.1%}, TP={tp}, FP={fp}, FN={fn}, TN={tn}")

import numpy as np
from sklearn.metrics import precision_recall_curve
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=200, n_features=5, random_state=42)

model = LogisticRegression()
model.fit(X, y)
y_prob = model.predict_proba(X)[:, 1]

precision, recall, thresholds = precision_recall_curve(y, y_prob)
f1_scores = 2 * (precision[:-1] * recall[:-1]) / (precision[:-1] + recall[:-1] + 1e-10)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]

print(f"Optimal threshold: {best_threshold:.4f}")
print(f"Best F1-score: {f1_scores[best_idx]:.4f}")

Now that we understand the theory behind Logistic Regression, it's time to build real models. In this section, we'll learn how to train Logistic Regression models using scikit-learn, understand the optimization process, evaluate model performance, and interpret results. You'll go from theory to practical implementation that you can use immediately.

Cost Function: Log Loss

To train a Logistic Regression model, we need to minimize a cost function. Linear Regression uses Mean Squared Error (MSE), but Logistic Regression uses Log Loss (also called binary cross-entropy). This cost function penalizes confident wrong predictions heavily while rewarding confident correct predictions.

# Understanding Log Loss: Why It Works
import numpy as np

def single_sample_loss(y_true, y_pred):
    """Calculate loss for a single sample"""
    epsilon = 1e-15  # Prevent log(0)
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    
    if y_true == 1:
        return -np.log(y_pred)
    else:
        return -np.log(1 - y_pred)

print("Log Loss Explained with Examples:")
print("="*60)

# Case 1: True label is 1 (positive class)
print("\nWhen TRUE LABEL = 1 (positive class):")
print("-"*40)
for pred in [0.99, 0.8, 0.5, 0.2, 0.01]:
    loss = single_sample_loss(1, pred)
    quality = "Excellent!" if loss < 0.1 else "Good" if loss < 0.5 else "Poor" if loss < 1 else "Terrible!"
    print(f"  Predict {pred:.2f} -> Loss: {loss:.4f} ({quality})")

# Case 2: True label is 0 (negative class)
print("\nWhen TRUE LABEL = 0 (negative class):")
print("-"*40)
for pred in [0.01, 0.2, 0.5, 0.8, 0.99]:
    loss = single_sample_loss(0, pred)
    quality = "Excellent!" if loss < 0.1 else "Good" if loss < 0.5 else "Poor" if loss < 1 else "Terrible!"
    print(f"  Predict {pred:.2f} -> Loss: {loss:.4f} ({quality})")

print("\nKey insight: Confident WRONG predictions are penalized heavily!")

Gradient Descent Optimization

Logistic Regression uses gradient descent to minimize the log loss. Gradient descent is an iterative algorithm that takes small steps toward the minimum of the cost function. The direction of each step is determined by the gradient (derivative) of the cost function.

import numpy as np

print("Gradient Descent - Mountain Analogy:")
print("="*50)

# Simplified 1D example: Finding minimum of f(x) = x^2
def f(x):
    return x ** 2

def gradient(x):
    return 2 * x  # Derivative of x^2

x = 5.0
learning_rate = 0.1

print(f"Start position: x = {x:.4f}, f(x) = {f(x):.4f}")
print("-"*50)

for step in range(10):
    grad = gradient(x)  # Check slope direction
    x = x - learning_rate * grad  # Move opposite to slope
    print(f"Step {step+1}: grad={grad:+.3f}, move to x={x:.4f}, f(x)={f(x):.4f}")

print("-"*50)
print(f"Final position: x = {x:.4f} (very close to 0!)")
print("\nIn Logistic Regression, we do the same but with multiple weights!")

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

def log_loss(y, y_pred):
    """Binary cross-entropy loss"""
    epsilon = 1e-15
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y * np.log(y_pred) + (1 - y) * np.log(1 - y_pred))

def gradient_descent_step(X, y, w, b, learning_rate):
    """One step of gradient descent"""
    m = X.shape[0]
    
    # Forward pass
    z = np.dot(X, w) + b
    y_pred = sigmoid(z)

    # Backward pass (gradients)
    dw = np.dot(X.T, (y_pred - y)) / m
    db = np.mean(y_pred - y)

    # Update weights
    w_new = w - learning_rate * dw
    b_new = b - learning_rate * db
    
    # Compute new loss
    z_new = np.dot(X, w_new) + b_new
    y_pred_new = sigmoid(z_new)
    loss = log_loss(y, y_pred_new)
    
    return w_new, b_new, loss

np.random.seed(42)
X = np.random.randn(50, 2)
y = (X[:, 0] + X[:, 1] > 0).astype(float)

w = np.random.randn(2) * 0.01
b = 0
learning_rate = 0.1
losses = []

for epoch in range(100):
    w, b, loss = gradient_descent_step(X, y, w, b, learning_rate)
    losses.append(loss)

plt.figure(figsize=(10, 5))
plt.plot(losses, linewidth=2)
plt.xlabel('Iteration')
plt.ylabel('Log Loss')
plt.title('Gradient Descent: Loss Decreasing Over Iterations')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print(f"Initial loss: {losses[0]:.4f}")
print(f"Final loss: {losses[-1]:.4f}")

Training with Scikit-learn

In practice, we don't implement gradient descent from scratch. Scikit-learn's LogisticRegression class handles everything for us. Let's train a real model on actual data.

from sklearn.linear_model import LogisticRegression
import numpy as np

hours_studied = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
passed_exam = np.array([0, 0, 0, 0, 1, 1, 1, 1])  # 0=fail, 1=pass

model = LogisticRegression()
model.fit(hours_studied, passed_exam)

test_hours = [[3.5], [4.5], [5.5]]
for hours in test_hours:
    prob = model.predict_proba(hours)[0]
    prediction = model.predict(hours)[0]
    result = "PASS" if prediction == 1 else "FAIL"
    print(f"Hours: {hours[0]} -> Probability: {prob[1]:.2%} -> {result}")

print(f"\nModel learned: coef={model.coef_[0][0]:.3f}, intercept={model.intercept_[0]:.3f}")

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = LogisticRegression(max_iter=10000)
model.fit(X_train_scaled, y_train)

y_train_pred = model.predict(X_train_scaled)
y_test_pred = model.predict(X_test_scaled)

y_train_proba = model.predict_proba(X_train_scaled)
y_test_proba = model.predict_proba(X_test_scaled)

print(f"Training accuracy: {model.score(X_train_scaled, y_train):.4f}")
print(f"Testing accuracy: {model.score(X_test_scaled, y_test):.4f}")
print(f"\nModel coefficients shape: {model.coef_.shape}")
print(f"First 5 coefficients: {model.coef_[0][:5]}")
print(f"Intercept: {model.intercept_[0]:.4f}")

Model Evaluation Metrics

Accuracy alone isn't enough to evaluate classification models, especially with imbalanced data. We need multiple metrics to understand different aspects of model performance.

y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]  # 4 spam, 6 not-spam
y_pred = [1, 1, 0, 0, 0, 0, 0, 0, 1, 0]  # Our model's predictions

tp = 2  # True Positive: Correctly predicted spam
fp = 1  # False Positive: Predicted spam but was not spam
fn = 2  # False Negative: Predicted not-spam but was spam
tn = 5  # True Negative: Correctly predicted not-spam

accuracy = (tp + tn) / (tp + tn + fp + fn)
print(f"\nAccuracy = (TP+TN)/(TP+TN+FP+FN)")
print(f"         = ({tp}+{tn})/({tp}+{tn}+{fp}+{fn}) = {accuracy:.2f}")
print(f"  Meaning: {accuracy*100:.0f}% of all predictions were correct")

precision = tp / (tp + fp)
print(f"\nPrecision = TP/(TP+FP) = {tp}/({tp}+{fp}) = {precision:.2f}")
print(f"  Meaning: {precision*100:.0f}% of predicted spam was actually spam")

recall = tp / (tp + fn)
print(f"\nRecall = TP/(TP+FN) = {tp}/({tp}+{fn}) = {recall:.2f}")
print(f"  Meaning: We caught {recall*100:.0f}% of all spam")

f1 = 2 * (precision * recall) / (precision + recall)
print(f"\nF1-Score = 2*(Precision*Recall)/(Precision+Recall) = {f1:.2f}")
print(f"  Meaning: Balanced score considering both precision and recall")

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (accuracy_score, precision_score, recall_score, 
                             f1_score, confusion_matrix, roc_auc_score, 
                             roc_curve, auc)
import matplotlib.pyplot as plt

X, y = make_classification(n_samples=1000, n_features=20, n_informative=10,
                           weights=[0.9, 0.1], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = LogisticRegression()
model.fit(X_train_scaled, y_train)

y_pred = model.predict(X_test_scaled)
y_proba = model.predict_proba(X_test_scaled)[:, 1]

print("Classification Metrics:")
print("=" * 40)
print(f"Accuracy:  {accuracy_score(y_test, y_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall:    {recall_score(y_test, y_pred):.4f}")
print(f"F1-Score:  {f1_score(y_test, y_pred):.4f}")
print(f"ROC-AUC:   {roc_auc_score(y_test, y_proba):.4f}")

tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
print("\nConfusion Matrix:")
print(f"TP: {tp}, FP: {fp}")
print(f"FN: {fn}, TN: {tn}")

fpr, tpr, _ = roc_curve(y_test, y_proba)
roc_auc = auc(fpr, tpr)

plt.figure(figsize=(10, 6))
plt.plot(fpr, tpr, color='b', lw=2, label=f'ROC curve (AUC = {roc_auc:.3f})')
plt.plot([0, 1], [0, 1], color='r', lw=2, linestyle='--', label='Random')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

Feature Importance

In Logistic Regression, the magnitude of coefficients indicates feature importance. Larger absolute coefficients mean the feature has more influence on the prediction. This makes Logistic Regression more interpretable than many other algorithms.

# Understanding Feature Importance in Logistic Regression
import numpy as np

# Suppose we're predicting if a customer will buy (1) or not (0)
# We have 3 features with these coefficients:
coefficients = {
    "visit_duration": 0.8,    # Time spent on website (minutes)
    "items_viewed": 1.5,      # Number of products viewed
    "previous_purchases": -0.3  # Already bought before (paradoxically negative!)
}

print("Feature Coefficients Explained:")
print("="*55)

for feature, coef in coefficients.items():
    direction = "increases" if coef > 0 else "decreases"
    impact = abs(coef)
    
    print(f"\n{feature}: {coef:+.2f}")
    print(f"  - Each unit increase {direction} log-odds by {impact:.2f}")
    print(f"  - Importance ranking: {'High' if impact > 1 else 'Medium' if impact > 0.5 else 'Low'}")

# Which feature matters most? Look at absolute values!
importance = {f: abs(c) for f, c in coefficients.items()}
sorted_features = sorted(importance.items(), key=lambda x: x[1], reverse=True)

print("\nFeature Importance (by absolute coefficient):")
for i, (feature, imp) in enumerate(sorted_features, 1):
    print(f"  {i}. {feature}: {imp:.2f}")

# Feature Importance from Coefficients
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
import pandas as pd
import matplotlib.pyplot as plt

# Load data
data = load_breast_cancer()
X = data.data
y = data.target
feature_names = data.feature_names

# Scale and train
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

model = LogisticRegression(max_iter=10000)
model.fit(X_scaled, y)

# Get feature importance (coefficients)
importance = pd.DataFrame({
    'Feature': feature_names,
    'Coefficient': model.coef_[0],
    'Abs_Coefficient': np.abs(model.coef_[0])
}).sort_values('Abs_Coefficient', ascending=False)

# Plot top 10 features
plt.figure(figsize=(10, 6))
top_features = importance.head(10)
colors = ['green' if c > 0 else 'red' for c in top_features['Coefficient']]
plt.barh(range(len(top_features)), top_features['Coefficient'], color=colors)
plt.yticks(range(len(top_features)), top_features['Feature'])
plt.xlabel('Coefficient Value')
plt.title('Top 10 Most Important Features')
plt.tight_layout()
plt.show()

print("\nTop 5 Most Important Features:")
print(importance.head())

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

iris = load_iris()
X, y = iris.data, iris.target

# Binary classification: use only classes 0 and 1
mask = y != 2
X, y = X[mask], y[mask]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression()
model.fit(X_train, y_train)

accuracy = model.score(X_test, y_test)
print(f"Accuracy: {accuracy:.4f}")

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import precision_score, recall_score

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = LogisticRegression()
model.fit(X_train_scaled, y_train)
y_proba = model.predict_proba(X_test_scaled)[:, 1]

print("Threshold | Precision | Recall")
print("-" * 35)
for threshold in [0.3, 0.4, 0.5, 0.6, 0.7]:
    y_pred = (y_proba >= threshold).astype(int)
    precision = precision_score(y_test, y_pred)
    recall = recall_score(y_test, y_pred)
    print(f"{threshold:.1f}      | {precision:.3f}     | {recall:.3f}")

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

param_grid = {
    'C': [0.001, 0.01, 0.1, 1, 10, 100],
    'penalty': ['l1', 'l2']
}

model = LogisticRegression(solver='liblinear', max_iter=10000)
grid_search = GridSearchCV(model, param_grid, cv=5, scoring='f1')
grid_search.fit(X_train_scaled, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.4f}")
print(f"Test score: {grid_search.score(X_test_scaled, y_test):.4f}")

Naive Bayes is one of the simplest yet surprisingly effective classification algorithms. Despite its "naive" assumption that all features are independent of each other (which is rarely true in real life), it often performs remarkably well, especially for text classification tasks like spam detection and sentiment analysis. The algorithm is based on Bayes' Theorem from probability theory, and its simplicity means it's incredibly fast to train and predict - perfect for large datasets or real-time applications.

Bayes' Theorem: The Foundation

At its core, Naive Bayes uses Bayes' Theorem to calculate the probability of a class given the observed features. The theorem states:

The "Naive" Assumption

The "naive" part comes from assuming that all features are conditionally independent given the class. This means we assume that knowing the value of one feature tells us nothing about other features (given the class). For example, in spam detection, we assume that the presence of the word "free" is independent of the word "money" - which is obviously not true! Yet this simplification makes the math tractable and often works surprisingly well in practice.

Implementing Gaussian Naive Bayes

Let's implement Gaussian Naive Bayes for classification on a standard dataset. This variant assumes features follow a normal distribution.

from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import numpy as np

# Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create and train Gaussian Naive Bayes classifier
gnb = GaussianNB()
gnb.fit(X_train, y_train)

# Make predictions
y_pred = gnb.predict(X_test)

# Evaluate the model
print("Gaussian Naive Bayes Results:")
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))

Text Classification with Multinomial Naive Bayes

Naive Bayes truly shines in text classification. Here's how to build a spam detector using Multinomial Naive Bayes:

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.pipeline import Pipeline

# Sample email data
emails = [
    "Get rich quick! Free money waiting for you!",
    "Meeting scheduled for tomorrow at 3pm",
    "Congratulations! You've won a million dollars!",
    "Please review the attached project proposal",
    "Limited time offer! Buy now and save 90%!",
    "Can we reschedule our call to next week?",
    "Claim your free prize now! Act immediately!",
    "The quarterly report is ready for your review"
]
labels = [1, 0, 1, 0, 1, 0, 1, 0]  # 1 = spam, 0 = not spam

# Create a pipeline with text vectorization and Naive Bayes
spam_detector = Pipeline([
    ('vectorizer', TfidfVectorizer(stop_words='english')),
    ('classifier', MultinomialNB(alpha=1.0))  # alpha is Laplace smoothing
])

# Train the model
spam_detector.fit(emails, labels)

# Test on new emails
new_emails = [
    "Free lottery tickets! Claim now!",
    "Let's discuss the project timeline"
]

predictions = spam_detector.predict(new_emails)
probabilities = spam_detector.predict_proba(new_emails)

for email, pred, prob in zip(new_emails, predictions, probabilities):
    label = "SPAM" if pred == 1 else "NOT SPAM"
    confidence = max(prob) * 100
    print(f"'{email[:40]}...' -> {label} ({confidence:.1f}% confident)")

from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

# Compare models using cross-validation
gnb = GaussianNB()
lr = LogisticRegression(max_iter=200)

gnb_scores = cross_val_score(gnb, X, y, cv=5)
lr_scores = cross_val_score(lr, X, y, cv=5)

print(f"Gaussian NB: {gnb_scores.mean():.4f} (+/- {gnb_scores.std():.4f})")
print(f"Logistic Regression: {lr_scores.mean():.4f} (+/- {lr_scores.std():.4f})")

from sklearn.naive_bayes import MultinomialNB
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

# Load subset of 20 newsgroups
categories = ['sci.space', 'rec.sport.hockey']
newsgroups = fetch_20newsgroups(subset='train', categories=categories)

alphas = [0.01, 0.1, 1.0, 10.0]

for alpha in alphas:
    pipeline = Pipeline([
        ('tfidf', TfidfVectorizer(max_features=5000)),
        ('nb', MultinomialNB(alpha=alpha))
    ])
    scores = cross_val_score(pipeline, newsgroups.data, newsgroups.target, cv=5)
    print(f"Alpha={alpha:5.2f}: {scores.mean():.4f} (+/- {scores.std():.4f})")

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# Sample movie reviews
reviews = [
    "This movie was absolutely fantastic! Great acting and plot.",
    "Terrible waste of time. Boring and predictable.",
    "I loved every minute of it. A masterpiece!",
    "Worst movie I've ever seen. Don't watch it.",
    "Brilliant performances by the entire cast. Highly recommend!",
    "So boring I fell asleep. Complete disappointment.",
    "An incredible journey with memorable characters.",
    "Awful script and terrible direction. Save your money."
]
sentiments = [1, 0, 1, 0, 1, 0, 1, 0]  # 1 = positive, 0 = negative

X_train, X_test, y_train, y_test = train_test_split(
    reviews, sentiments, test_size=0.25, random_state=42
)

# Create pipeline
sentiment_model = Pipeline([
    ('tfidf', TfidfVectorizer(ngram_range=(1, 2), max_features=1000)),
    ('nb', MultinomialNB())
])

sentiment_model.fit(X_train, y_train)
predictions = sentiment_model.predict(X_test)

print("Sentiment Analysis Results:")
print(classification_report(y_test, predictions, target_names=['Negative', 'Positive']))

K-Nearest Neighbors is one of the most intuitive machine learning algorithms - it literally says "tell me who your neighbors are, and I'll tell you who you are." When classifying a new data point, KNN looks at the K closest training examples and assigns the class that appears most frequently among those neighbors. It's a "lazy learner" because it doesn't build a model during training; instead, it stores all the training data and does the work at prediction time. This makes it simple to understand and implement, but potentially slow for large datasets.

How KNN Works

Choosing the Right K

The choice of K (number of neighbors) dramatically affects the model's behavior:

Distance Metrics

The "closeness" of neighbors depends on how we measure distance. Different metrics work better for different types of data:

Implementing KNN

from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report
import numpy as np

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# IMPORTANT: Scale features for KNN (distance-based algorithm)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create and train KNN classifier
knn = KNeighborsClassifier(n_neighbors=5, metric='euclidean')
knn.fit(X_train_scaled, y_train)

# Predict and evaluate
y_pred = knn.predict(X_test_scaled)
print(f"KNN Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))

Finding the Optimal K

import matplotlib.pyplot as plt

# Test different K values
k_values = range(1, 31)
train_scores = []
test_scores = []

for k in k_values:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train_scaled, y_train)
    train_scores.append(knn.score(X_train_scaled, y_train))
    test_scores.append(knn.score(X_test_scaled, y_test))

# Plot results
plt.figure(figsize=(10, 6))
plt.plot(k_values, train_scores, 'b-o', label='Training Accuracy')
plt.plot(k_values, test_scores, 'r-s', label='Test Accuracy')
plt.xlabel('Number of Neighbors (K)')
plt.ylabel('Accuracy')
plt.title('KNN: Finding the Optimal K')
plt.legend()
plt.grid(True, alpha=0.3)

# Find best K
best_k = k_values[np.argmax(test_scores)]
print(f"Best K: {best_k} with test accuracy: {max(test_scores):.4f}")

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

# Without scaling
knn_no_scale = KNeighborsClassifier(n_neighbors=5)
scores_no_scale = cross_val_score(knn_no_scale, X, y, cv=5)

# With scaling
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
knn_scaled = KNeighborsClassifier(n_neighbors=5)
scores_scaled = cross_val_score(knn_scaled, X_scaled, y, cv=5)

print(f"Without scaling: {scores_no_scale.mean():.4f}")
print(f"With scaling: {scores_scaled.mean():.4f}")

from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler

wine = load_wine()
X = StandardScaler().fit_transform(wine.data)
y = wine.target

metrics = ['euclidean', 'manhattan', 'chebyshev']

for metric in metrics:
    knn = KNeighborsClassifier(n_neighbors=5, metric=metric)
    scores = cross_val_score(knn, X, y, cv=5)
    print(f"{metric:12s}: {scores.mean():.4f} (+/- {scores.std():.4f})")

from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler

# Create dataset with some overlap
X, y = make_classification(n_samples=500, n_features=10, 
                          n_informative=5, n_redundant=2,
                          n_clusters_per_class=2, random_state=42)
X = StandardScaler().fit_transform(X)

for k in [3, 7, 15]:
    knn_uniform = KNeighborsClassifier(n_neighbors=k, weights='uniform')
    knn_distance = KNeighborsClassifier(n_neighbors=k, weights='distance')
    
    scores_uniform = cross_val_score(knn_uniform, X, y, cv=5).mean()
    scores_distance = cross_val_score(knn_distance, X, y, cv=5).mean()
    
    print(f"K={k:2d}: Uniform={scores_uniform:.4f}, Distance={scores_distance:.4f}")

Many real-world problems have more than two classes: classifying emails into spam/promotional/personal/social, recognizing handwritten digits (0-9), or identifying plant species. While some algorithms like Decision Trees and KNN naturally handle multiple classes, others like Logistic Regression and SVM are inherently binary classifiers. In this section, we'll explore strategies to extend binary classifiers to multi-class problems, and how to use algorithms that natively support multiple classes.

One-vs-Rest (OvR) Strategy

One-vs-One (OvO) Strategy

Multinomial (Softmax) Logistic Regression

Instead of using multiple binary classifiers, we can extend Logistic Regression to directly predict probabilities across all classes using the softmax function. This is more elegant and often more effective:

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X, y = iris.data, iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# OvR (One-vs-Rest) strategy
lr_ovr = LogisticRegression(multi_class='ovr', max_iter=200)
lr_ovr.fit(X_train, y_train)
print(f"One-vs-Rest Accuracy: {lr_ovr.score(X_test, y_test):.4f}")

# Multinomial (Softmax) - native multi-class
lr_multinomial = LogisticRegression(multi_class='multinomial', max_iter=200)
lr_multinomial.fit(X_train, y_train)
print(f"Multinomial Accuracy: {lr_multinomial.score(X_test, y_test):.4f}")

# Compare probability outputs
sample = X_test[0:1]
print(f"\nProbabilities for first sample:")
print(f"OvR: {lr_ovr.predict_proba(sample)[0]}")
print(f"Multinomial: {lr_multinomial.predict_proba(sample)[0]}")

Multi-class with Different Algorithms

from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler

# Load digit recognition dataset (10 classes: 0-9)
digits = load_digits()
X = StandardScaler().fit_transform(digits.data)
y = digits.target

print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features, {len(set(y))} classes\n")

# Compare different multi-class approaches
models = {
    'Logistic (OvR)': LogisticRegression(multi_class='ovr', max_iter=1000),
    'Logistic (Multinomial)': LogisticRegression(multi_class='multinomial', max_iter=1000),
    'Naive Bayes': GaussianNB(),
    'KNN (k=5)': KNeighborsClassifier(n_neighbors=5),
    'Decision Tree': DecisionTreeClassifier(max_depth=10),
    'SVM (OvO)': SVC(kernel='rbf')
}

for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5)
    print(f"{name:25s}: {scores.mean():.4f} (+/- {scores.std():.4f})")

Explicit OvR and OvO Wrappers

from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

# Explicitly wrap a binary classifier for multi-class
# One-vs-Rest wrapper
ovr_classifier = OneVsRestClassifier(LogisticRegression(max_iter=1000))
ovr_classifier.fit(X_train, y_train)
print(f"Explicit OvR: {ovr_classifier.score(X_test, y_test):.4f}")
print(f"Number of classifiers: {len(ovr_classifier.estimators_)}")

# One-vs-One wrapper
ovo_classifier = OneVsOneClassifier(SVC(kernel='linear'))
ovo_classifier.fit(X_train, y_train)
print(f"Explicit OvO: {ovo_classifier.score(X_test, y_test):.4f}")
print(f"Number of classifiers: {len(ovo_classifier.estimators_)}")

from sklearn.multiclass import OneVsOneClassifier
from sklearn.svm import SVC
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.data, digits.target

# Calculate theoretically
n_classes = len(set(y))
n_classifiers = n_classes * (n_classes - 1) // 2
print(f"Theoretical: {n_classes} classes -> {n_classifiers} classifiers")

# Verify with code
ovo = OneVsOneClassifier(SVC())
ovo.fit(X, y)
print(f"Actual: {len(ovo.estimators_)} classifiers")

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
import time

digits = load_digits()
X, y = digits.data, digits.target

for strategy in ['ovr', 'multinomial']:
    model = LogisticRegression(multi_class=strategy, max_iter=5000, solver='lbfgs')
    
    start = time.time()
    scores = cross_val_score(model, X, y, cv=5)
    elapsed = time.time() - start
    
    print(f"{strategy:12s}: Accuracy={scores.mean():.4f}, Time={elapsed:.2f}s")

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target, test_size=0.3, random_state=42
)

model = LogisticRegression(max_iter=5000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Create confusion matrix
cm = confusion_matrix(y_test, y_pred)

plt.figure(figsize=(10, 8))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
            xticklabels=digits.target_names,
            yticklabels=digits.target_names)
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Multi-class Confusion Matrix: Digit Recognition')
plt.show()

# Find most confused pairs
import numpy as np
np.fill_diagonal(cm, 0)  # Ignore correct predictions
max_idx = np.unravel_index(cm.argmax(), cm.shape)
print(f"Most confused: {max_idx[0]} misclassified as {max_idx[1]} ({cm[max_idx]} times)")