Monitor Model Drift
See Extended Examples for complete configuration files and templates.
Detect and alert on data drift and concept drift in production ML models using statistical tests and automated monitoring.
When to Use
- Production ML models experiencing unexplained performance degradation
- New data distributions differ from training data
- Seasonal or temporal shifts in input features
- Need proactive alerts before business metrics are impacted
- Regulatory requirements for model monitoring (e.g., SR 11-7, EU AI Act)
- Multiple model versions deployed requiring drift comparison
Inputs
- Required: Production model predictions and features (last 30-90 days)
- Required: Reference dataset (training or validation data)
- Required: Ground truth labels (may be delayed)
- Optional: Feature importance scores or SHAP values
- Optional: Business metric thresholds for alerting
- Optional: Historical drift reports for trend analysis
Procedure
Step 1: Install and Configure Evidently AI
Set up the monitoring framework with appropriate dependencies.
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install Evidently and dependencies
pip install evidently pandas scikit-learn prometheus-client
# Create monitoring directory structure
mkdir -p monitoring/{reports,config,alerts}
Create configuration file:
# monitoring/config/drift_config.py
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
from evidently.metrics import (
DatasetDriftMetric,
DatasetMissingValuesMetric,
ColumnDriftMetric,
)
# ... (see EXAMPLES.md for complete implementation)
Expected: Configuration file created with thresholds matching your model's tolerance.
On failure: Start with conservative thresholds (PSI > 0.2, KS p-value < 0.01) and tune based on false positive rate.
Step 2: Implement Data Drift Detection
Create drift detection pipeline with multiple statistical tests.
# monitoring/drift_detector.py
import pandas as pd
import numpy as np
from scipy.stats import ks_2samp, chi2_contingency
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
from evidently.metrics import ColumnDriftMetric, DatasetDriftMetric
from datetime import datetime, timedelta
# ... (see EXAMPLES.md for complete implementation)
Expected: Drift detection runs successfully, produces JSON report with per-feature statistics, and identifies drifted features.
On failure: Check for missing values (impute or drop), ensure reference and current data have same columns, verify data types match between datasets.
Step 3: Generate Evidently Reports
Create visual HTML reports for human review and debugging.
# monitoring/generate_reports.py
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
from evidently.metrics import (
ColumnDriftMetric,
DatasetDriftMetric,
DatasetMissingValuesMetric,
)
# ... (see EXAMPLES.md for complete implementation)
Expected: HTML reports generated in monitoring/reports/, viewable in browser with interactive charts showing distribution comparisons.
On failure: Verify write permissions to output directory, check that Evidently version is >= 0.4.0, ensure data frames have sufficient rows (>100 recommended).
Step 4: Implement Concept Drift Detection
Monitor prediction performance to detect concept drift (relationship between features and target changes).
# monitoring/concept_drift.py
import pandas as pd
import numpy as np
from sklearn.metrics import roc_auc_score, mean_squared_error, accuracy_score
from typing import Dict, List
import json
# ... (see EXAMPLES.md for complete implementation)
Expected: Performance monitoring detects when model accuracy/AUC drops below threshold, signaling potential concept drift.
On failure: Ensure ground truth labels are available (may require delayed validation batch job), verify prediction scores are properly calibrated (0-1 range for classification), check for label leakage in features.
Step 5: Set Up Automated Alerting
Integrate drift detection with alerting systems (Slack, PagerDuty, email).
# monitoring/alerting.py
import requests
import json
from typing import Dict, List
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)
Expected: Alerts sent to Slack/PagerDuty when drift detected, with severity based on drift share and critical feature involvement.
On failure: Test webhook URLs with curl first, verify PagerDuty integration key has correct permissions, check firewall rules for outbound HTTPS, implement retry logic for transient network failures.
Step 6: Schedule Monitoring Jobs
Automate drift detection to run on schedule (daily or weekly).
# monitoring/scheduler.py
import schedule
import time
import logging
from datetime import datetime, timedelta
import pandas as pd
logging.basicConfig(
# ... (see EXAMPLES.md for complete implementation)
Alternatively, use cron:
# Add to crontab (crontab -e)
# Run daily at 2 AM
0 2 * * * cd /path/to/monitoring && /path/to/venv/bin/python scheduler.py >> logs/cron.log 2>&1
Or use Airflow DAG:
# airflow/dags/drift_monitoring_dag.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'ml-team',
'depends_on_past': False,
# ... (see EXAMPLES.md for complete implementation)
Expected: Monitoring runs automatically on schedule, generates reports, sends alerts only when drift exceeds thresholds, logs all activity.
On failure: Check scheduler process is running (ps aux | grep scheduler), verify cron service is active, ensure data sources are accessible, review logs for exceptions, set up dead man's switch alert if job doesn't run.
Validation
Common Pitfalls
- Stale reference data: Update reference dataset quarterly or after model retraining to reflect natural data evolution
- Sample size mismatch: Ensure current and reference datasets have similar sizes (>1000 rows each) for reliable statistics
- Missing ground truth: Concept drift requires labels; implement delayed labeling pipeline if real-time labels unavailable
- Seasonality confusion: Weekly/monthly patterns may trigger false positives; use time-aligned reference windows or deseasonalize features
- Alert fatigue: Start with high thresholds and gradually lower based on actual model retraining cadence
- Ignoring data quality drift: Monitor missing values, outliers, and encoding errors separately from distribution drift
- Over-reliance on aggregate metrics: Per-feature analysis crucial; aggregate drift may mask critical individual feature shifts
- Neglecting prediction distribution: Even without ground truth, sudden prediction distribution shifts signal issues
Related Skills
detect-anomalies-aiops - Time series anomaly detection for operational metricsdeploy-ml-model-serving - Model deployment patterns and versioningsetup-prometheus-monitoring - Infrastructure metrics collectionreview-data-analysis - Statistical analysis validation and peer review