A Guide to Debug your ML Model

Introduction

When a model fails to meet expectations, the instinct is often to tweak hyperparameters or switch architectures. However, experience from building production systems at KishnaKushwaha shows that the majority of issues stem from data quality or pipeline bugs. Recognizing where to look first saves hours of fruitless experimentation.

We will cover a structured checklist: verify data integrity, detect leakage, assess class imbalance, inspect missing values, and then move on to model‑centric diagnostics such as loss curves, overfitting signs, and underfitting signals. Each step includes practical code snippets you can drop into a notebook. If you're exploring this area, check out AI interview prep tool — Try Intervu free.

The techniques described here have been applied to improve the predictive maintenance models in GrowthAI, the resume‑matching engine in Intervu, and the intent classification module of the Voice Agent.

Why It Matters

A model that appears to perform well in validation but collapses in production often suffers from subtle data leakage or preprocessing mismatches. These issues erode trust and can lead to costly business decisions based on faulty predictions.

By systematically checking for NaNs, outliers, and leakage before tuning hyperparameters, you ensure that any performance gains are genuine. This approach has reduced debugging cycles by roughly 40 % in the projects led by Kishna.

Furthermore, understanding whether a model is overfitting or underfitting guides you toward the right remedy—regularization versus increased capacity—saving compute resources and avoiding wasted experiments.

Core Concepts

Before diving into code, it helps to define the key phenomena we will hunt for.

• Data leakage: Information from the future or the target inadvertently becomes a feature, inflating validation metrics.
• Missing values: NaNs that propagate through calculations can produce NaN predictions or break gradient flow.
• Class imbalance: One label dominates the dataset, allowing a model to achieve high accuracy by always predicting the majority class.
• Outliers: Extreme values can skew loss surfaces, especially for distance‑based or linear models.
• Overfitting: Training accuracy far exceeds test accuracy, indicating memorization.
• Underfitting: Both training and test accuracies remain low, suggesting insufficient model capacity or inadequate training.

Each concept has a corresponding diagnostic tool, which we will explore next.

Architecture and How It Works

The debugging workflow follows a linear pipeline that can be iterated as new issues surface.

This structure mirrors the validation strategy used in the Voice Agent project, where a strict temporal split prevented leakage of future utterance features.

Step‑by‑Step Implementation

Below is a concrete implementation using Python, pandas, and scikit‑learn. Feel free to adapt the snippet to PyTorch or TensorFlow workflows.

# 1. Load data
import pandas as pd
df = pd.read_csv('/assets/datasets/telemetry.csv')

# 2. Separate features and target
X = df.drop(columns=['failure'])
y = df['failure']

# 3. Split BEFORE any preprocessing
from sklearn.model_selection import train_test_split
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp)  # 0.25 × 0.8 = 0.2 validation

# 4. Check for NaNs in the training set
def report_nans(dataframe):
    nan_counts = dataframe.isnull().sum()
    total = len(dataframe)
    nan_pct = (nan_counts / total) * 100
    return nan_counts[nan_counts > 0], nan_pct[nan_pct > 0]

nan_cols, nan_pct = report_nans(X_train)
print('Columns with NaNs:', nan_cols.to_dict())
print('Percentage missing:', nan_pct.to_dict())

# 5. Simple imputation (median) fitted on train only
from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy='median')
X_train_imp = imp.fit_transform(X_train)
X_val_imp = imp.transform(X_val)
X_test_imp = imp.transform(X_test)

# 6. Scale features (standardization) fitted on train
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_imp)
X_val_scaled = scaler.transform(X_val_imp)
X_test_scaled = scaler.transform(X_test_imp)

# 7. Baseline model
from sklearn.linear_model import LogisticRegression
baseline = LogisticRegression(max_iter=1000, n_jobs=-1)
baseline.fit(X_train_scaled, y_train)
print('Baseline val accuracy:', baseline.score(X_val_scaled, y_val))

# 8. Placeholder for your complex model (e.g., a neural net)
#    Train it, then compare validation/test scores to the baseline.

This script guarantees that no information from the validation or test sets leaks into the preprocessing steps, a mistake that once caused a 15 % overestimation of accuracy in the GrowthAI forecasting module.

After confirming that the baseline outperforms a constant predictor, you can proceed to train your main architecture while watching for the warning signs discussed earlier.

Practical Examples

Example 1: Detecting and Handling Missing Values

Suppose you notice that the column ‘sensor_3’ has 12 % missing entries. Instead of dropping the column, you can impute using the median value computed solely from the training split, as shown in the snippet above. This preserves the feature’s distribution while avoiding leakage.

Visual inspection of a histogram before and after imputation can reveal whether the imputed values create an artificial spike; if they do, consider using a more sophisticated method like iterative imputation.

Example 2: Spotting Data Leakage Through Temporal Features

In a predictive maintenance scenario, you might inadvertently include a feature like ‘days_since_last_service’ that is calculated using the failure timestamp itself. To audit, compute the correlation between each feature and the target in the training set; an unusually high absolute correlation (>0.9) warrants investigation.

One effective test is to train a model on a single batch (see Example 3) and verify that loss drops to near zero. If it does not, the problem may be a broken pipeline rather than the model architecture.

Example 3: Single‑Batch Overfit Test

The following PyTorch snippet grabs one batch of data and trains the model for many epochs. If the architecture and learning rate are suitable, the loss should approach zero.

import torch
import torch.nn as nn
import torch.optim as optim

# Assume `train_loader` yields batches of tensors
model = MyModel()               # replace with your architecture
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

# Grab a single batch
batch_x, batch_y = next(iter(train_loader))
model.train()
for epoch in range(200):
    optimizer.zero_grad()
    outputs = model(batch_x)
    loss = criterion(outputs, batch_y)
    loss.backward()
    optimizer.step()
    if epoch % 20 == 0:
        print(f'Epoch {epoch}, loss: {loss.item():.6f}')
# After the loop, loss should be close to 0 if the model can learn

If the loss remains flat, check the learning rate (too high or too low), verify that gradients are not vanishing, and ensure the loss function matches the task (e.g., use BCEWithLogitsLoss for binary classification). This test saved the Intervu team from weeks of tuning a transformer whose embedding layer was accidentally set to non‑trainable.

Frequently Asked Questions (FAQs)

Conclusion

Debugging machine learning models is less about random tweaks and more about a disciplined, evidence‑based process. By first eliminating data‑centric pitfalls—leakage, missing values, outliers, and class imbalance—you create a trustworthy foundation for model training. Subsequent checks on loss curves, single‑batch learnability, and bias‑variance trade‑offs guide you toward the right architectural or regularization adjustments.

Applying this workflow has consistently improved the reliability of systems such as GrowthAI’s anomaly detector, Intervu’s skill‑matching classifier, and the Voice Agent’s intent parser. Remember: the model is only as good as the data it learns from and the rigor with which you validate each step. Keep a notebook of your debugging experiments; over time, you’ll develop an intuition that turns frustrating failures into rapid insights.