deepchecks-tests
Deepchecks is "a holistic open-source solution for all of your AI &
ML validation needs" per the Deepchecks welcome. Validates data
integrity, train-test splits, model evaluation, end-to-end model
development from research through production.
When to use
- Pre-training: validate data integrity + train-test leakage.
- Pre-deployment: model evaluation suite as merge gate.
- Production monitoring: same suite re-run on production samples
to detect drift.
Step 1 - Install
pip install deepchecks
Per the Deepchecks welcome page.
Step 2 - Wrap data
For tabular models:
from deepchecks.tabular import Dataset
train_ds = Dataset(
train_df,
label="target",
cat_features=CATEGORICAL_COLUMNS,
)
test_ds = Dataset(
test_df,
label="target",
cat_features=CATEGORICAL_COLUMNS,
)
cat_features matters for distribution checks. The Vision and NLP
APIs differ - see the Deepchecks welcome section linking
quickstarts for each data type.
Step 3 - Run the data integrity suite
from deepchecks.tabular.suites import data_integrity
integrity = data_integrity()
result = integrity.run(train_ds)
result.save_as_html("data_integrity.html")
Catches: duplicate rows, missing values, mixed types, conflicting
labels, single-value features, string mismatches.
Step 4 - Run the train-test validation suite
from deepchecks.tabular.suites import train_test_validation
validation = train_test_validation()
result = validation.run(train_ds, test_ds)
result.save_as_html("train_test_validation.html")
Catches: target drift, feature drift, train-test data leakage, label
imbalance, dataset size mismatch.
Step 5 - Run the model evaluation suite
from deepchecks.tabular.suites import model_evaluation
evaluation = model_evaluation()
result = evaluation.run(train_ds, test_ds, model)
result.save_as_html("model_evaluation.html")
if not result.passed_conditions():
raise SystemExit("Deepchecks model evaluation failed")
Catches: performance regression vs baseline, weak segments, calibration
issues, prediction drift between train and test.
Step 6 - Per-check thresholds
from deepchecks.tabular.checks import FeatureDrift
check = FeatureDrift().add_condition_drift_score_less_than(
max_allowed_categorical_score=0.2,
max_allowed_numeric_score=0.1,
)
result = check.run(train_ds, test_ds)
if not result.passed_conditions():
print(result.value)
raise SystemExit("FeatureDrift failed threshold")
Each check has add_condition_* methods; chain them for per-check
gating.
Step 7 - CI integration
- name: Deepchecks suite
run: |
python ml/deepchecks_suite.py
- name: Upload Deepchecks reports
uses: actions/upload-artifact@v4
with:
name: deepchecks-reports
path: "*.html"
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
Skip cat_features | All categorical checks misfire | Always specify (Step 2) |
Skip data_integrity suite | Train on leaky / dup-heavy data | Run before train_test_validation (Step 3) |
| Block CI on every check | Hundreds of warning conditions; team disables | Define per-check thresholds (Step 6); gate Critical only |
| Re-run on the SAME test split each PR | Fixed split → fixed results; no drift detection | Use rolling/cross-validation splits |
| Reuse training data as "current" for production monitoring | Always passes drift; blind to real drift | Use real production samples |
Limitations
- Deepchecks' "default conditions" are tuned for textbook ML; they
may be too strict for real-world skewed data. Tune via
add_condition_* methods (Step 6).
- Vision suite memory footprint: large image datasets need batched
runs. See the Deepchecks welcome vision quickstart link.
References
- Deepchecks welcome - overview, install, suite/check architecture,
per-data-type quickstart links