ML Fundamentals Practice Problems

Your Task

Create a Jupyter Notebook called ml_pipeline.ipynb that implements a complete ML data preprocessing and evaluation pipeline. Your code must load the dataset, perform exploratory data analysis, preprocess features, split data properly, and calculate evaluation metrics on a baseline model.

You will work with an Employee Attrition dataset. Create this CSV file as shown below:

File: employees.csv (Employee Data)

employee_id,age,gender,department,job_role,years_at_company,monthly_income,satisfaction_score,work_life_balance,overtime,distance_from_home,education_level,performance_rating,attrition
E001,32,Male,Engineering,Software Engineer,5,85000,4.2,3,Yes,12,Bachelor,4,No
E002,28,Female,Sales,Sales Rep,2,45000,2.8,2,Yes,25,Bachelor,3,Yes
E003,45,Male,HR,HR Manager,12,95000,4.5,4,No,5,Master,5,No
E004,35,Female,Engineering,Data Scientist,3,92000,3.5,3,Yes,18,PhD,4,No
E005,26,Male,Marketing,Marketing Analyst,1,48000,2.2,2,Yes,30,Bachelor,2,Yes
E006,41,Female,Finance,Financial Analyst,8,78000,4.0,4,No,8,Master,4,No
E007,29,Male,Engineering,DevOps Engineer,4,82000,3.8,3,No,15,Bachelor,4,No
E008,38,Female,Sales,Sales Manager,7,88000,3.2,2,Yes,22,Master,3,Yes
E009,24,Male,Marketing,Content Writer,1,42000,2.5,2,Yes,35,Bachelor,3,Yes
E010,52,Female,HR,HR Director,15,125000,4.8,5,No,3,Master,5,No
E011,31,Male,Engineering,Frontend Dev,3,75000,3.0,3,Yes,20,Bachelor,3,No
E012,27,Female,Finance,Accountant,2,55000,2.9,2,Yes,28,Bachelor,3,Yes
E013,44,Male,Engineering,Tech Lead,10,115000,4.4,4,No,7,Master,5,No
E014,33,Female,Sales,Sales Rep,4,52000,3.3,3,No,16,Bachelor,3,No
E015,36,Male,Marketing,Marketing Manager,6,85000,4.1,4,No,10,Master,4,No
E016,25,Female,Engineering,Junior Dev,1,55000,2.6,2,Yes,32,Bachelor,2,Yes
E017,48,Male,Finance,CFO,14,150000,4.7,5,No,4,MBA,5,No
E018,30,Female,HR,Recruiter,3,52000,3.4,3,Yes,19,Bachelor,3,No
E019,34,Male,Engineering,Backend Dev,5,88000,3.9,3,No,14,Master,4,No
E020,22,Female,Marketing,Intern,0,28000,2.0,1,Yes,40,High School,2,Yes

Your ml_pipeline.ipynb must implement ALL of the following functions. Each function is mandatory and will be tested individually.

def load_and_explore(filename):
    """Load dataset and return exploration summary."""
    # Must return: dict with 'shape', 'dtypes', 'missing', 'stats'
    pass

def visualize_distributions(df):
    """Create and save EDA visualizations."""
    # Must save: eda_plots.png
    pass

def handle_missing_values(df, strategy='auto'):
    """Handle missing values with specified strategy."""
    # Return: (cleaned_df, missing_report_dict)
    pass

def handle_outliers(df, columns, method='iqr'):
    """Detect and handle outliers using IQR method."""
    # Return: (cleaned_df, outliers_report_dict)
    pass

def encode_categorical(df, columns):
    """Encode categorical variables appropriately."""
    # Return: (encoded_df, encoders_dict)
    pass

def scale_features(df, columns, method='standard'):
    """Scale numerical features using specified method."""
    # Return: (scaled_df, scaler_object)
    pass

def split_data(df, target_column, test_size=0.2, stratify=True):
    """Split data into train and test sets with stratification."""
    # Return: (X_train, X_test, y_train, y_test)
    pass

def get_correlation_matrix(df, threshold=0.7):
    """Generate correlation matrix and identify high correlations."""
    # Return: list of (col1, col2, correlation) tuples
    pass

def calculate_metrics(y_true, y_pred):
    """Calculate classification evaluation metrics."""
    # Return: dict with 'accuracy', 'precision', 'recall', 'f1', 'confusion_matrix'
    pass

def train_baseline(X_train, X_test, y_train, y_test):
    """Train a baseline logistic regression model."""
    # Return: (model, y_pred)
    pass

def save_processed_data(X_train, X_test, y_train, y_test, prefix='processed'):
    """Save preprocessed train and test data to CSV files."""
    # Return: list of saved file paths
    pass

def main():
    # 1. Load and explore data
    df = pd.read_csv("employees.csv")
    exploration = load_and_explore("employees.csv")
    
    # 2. Visualize distributions
    visualize_distributions(df)
    
    # 3. Handle missing values
    df_clean, missing_report = handle_missing_values(df)
    
    # 4. Handle outliers
    numerical_cols = ['age', 'monthly_income', 'distance_from_home']
    df_clean, outlier_report = handle_outliers(df_clean, numerical_cols)
    
    # 5. Encode categorical variables
    cat_cols = ['gender', 'department', 'overtime', 'education_level']
    df_encoded, encoders = encode_categorical(df_clean, cat_cols)
    
    # 6. Scale features
    scale_cols = ['age', 'years_at_company', 'monthly_income', 'distance_from_home']
    df_scaled, scaler = scale_features(df_encoded, scale_cols)
    
    # 7. Split data
    X_train, X_test, y_train, y_test = split_data(df_scaled, 'attrition')
    
    # 8. Get correlations
    correlations = get_correlation_matrix(df_scaled)
    
    # 9. Train baseline model
    model, y_pred = train_baseline(X_train, X_test, y_train, y_test)
    
    # 10. Calculate metrics
    metrics = calculate_metrics(y_test, y_pred)
    print(f"Baseline Model Metrics: {metrics}")
    
    # 11. Save processed data
    save_processed_data(X_train, X_test, y_train, y_test)
    
    print("ML Pipeline Complete!")

if __name__ == "__main__":
    main()

Create a public GitHub repository with the exact name shown below:

Required Files

employee-attrition-pipeline/
├── ml_pipeline.ipynb         # Your Jupyter Notebook with ALL 12 functions
├── employees.csv             # Input dataset (as provided or extended)
├── eda_plots.png             # Generated EDA visualizations
├── correlation_matrix.png    # Generated correlation heatmap
├── processed_X_train.csv     # Preprocessed training features
├── processed_X_test.csv      # Preprocessed test features
├── processed_y_train.csv     # Training labels
├── processed_y_test.csv      # Test labels
└── README.md                 # REQUIRED - see contents below

Your assignment will be graded on the following criteria: