本文へ移動
cccskills
無料GitHub で公開

diagnosing-ml-failures

Isolate the root cause of ML performance drops, inconsistent evaluations, prediction errors, and training-serving mismatches across data, labels, splits, pipelines, models, metrics, and runtime behavior. Use when investigating a reproducible failure or regression, not routine model selection or general performance validation.

インストール方法を見る

含まれるファイル(4)

  • SKILL.md4.5 KB
  • agents/openai.yaml236 B
  • assets/failure-investigation-log.md2.8 KB
  • references/failure-isolation.md4.1 KB

SKILL.md(原文)

インストールする前に、エージェントに与えられる指示の中身を確認できます。

Diagnosing ML Failures

Turn a vague failure into a minimal, testable discrepancy and isolate the first layer where expected and observed behavior diverge. Do not begin with hyperparameter tuning: it can hide the fault while destroying causal evidence.

Define the symptom

Capture one concrete failing example or comparison:

  • expected versus observed behavior;
  • first known bad and last known good run, version, or time window;
  • affected population and unaffected control population;
  • metric implementation, threshold, and aggregation level;
  • exact input, artifact, code revision, configuration, and environment when available.

Reproduce the symptom through the smallest stable entry point. If it is intermittent, estimate frequency and identify what varies between runs before changing the system.

Isolate the first broken layer

Trace the same records through boundaries in causal order:

  1. Source and semantics: extraction window, identifiers, units, timestamps, schema, missingness, duplicates, and joins.
  2. Ground truth: label definition, observation window, reviewer process, label delay, drift, and adjudication.
  3. Split and sampling: entity overlap, temporal leakage, cohort shift, weighting, and deduplication.
  4. Feature pipeline: availability time, fit/transform state, ordering, encoding, defaults, and training-serving parity.
  5. Model artifact: weights, feature contract, class order, preprocessing bundle, and artifact version.
  6. Metric and decision logic: denominator, grouping, threshold, calibration, exclusions, and business-cost mapping.
  7. Runtime and delivery: dependency versions, hardware nondeterminism, caching, serialization, concurrency, and response postprocessing.

Compare invariants at each boundary - row counts, key uniqueness, hashes, distributions, schemas, representative record traces - and stop at the earliest divergence. Downstream differences may be consequences, not causes.

Test competing hypotheses

Maintain multiple plausible hypotheses until evidence distinguishes them. For each hypothesis, define a discriminating test whose outcomes differ between explanations. Prefer controlled swaps and minimal counterfactuals:

  • old data with new code versus new data with old code;
  • frozen artifact through offline and online pipelines;
  • known-good records before and after one transformation;
  • recomputed metric from raw predictions versus reported metric;
  • independently adjudicated error sample versus existing labels.

Change one explanatory variable at a time where practical. Preserve all failed tests and negative results in a rejected-hypothesis log; they prevent repeated work and constrain future explanations.

Treat ground truth as a suspect, not an oracle

When apparent model failures may be annotation errors or ambiguous cases, sample errors without cherry-picking. Obtain independent human review only when an authorized reviewer and an approved data-sharing path are available; otherwise specify the review needed without transmitting records. Keep original labels, reviewer decisions, disagreements, and adjudication separate. A corrected label may explain an example without proving the model is correct for the population.

Close with proof, not plausibility

A diagnosis is complete only when the proposed cause predicts the symptom and an authorized, isolated, reversible intervention removes it without breaking an appropriate control. Verify at the same scope as the original failure: fixing one example does not prove a population regression is resolved. Diagnose and propose a fix by default; modify or deploy the real system only when the user requested that additional action.

Use assets/failure-investigation-log.md to preserve observations, experiments, rejected hypotheses, root-cause evidence, and regression coverage. For detailed layer probes and controlled isolation patterns, read references/failure-isolation.md.

If several layers remain plausible, report the narrowed boundary, remaining hypotheses, and next discriminating test. Do not label the most intuitive explanation as root cause.

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

同じリポジトリのスキル

概要と使いどころ

Audit datasets, joins, labels, and ground truth before analysis or modeling. Use when data meaning, row grain, time semantics, source-of-truth reliability, or label construction may invalidate conclusions; not for general model evaluation after the evidence base is already trusted.

日本語の概要は準備中です。原文の説明を表示しています。

aiopshwang/data-analysis-ml-agent-skills122026年8月27日 更新

Design leakage-safe machine learning experiments that mirror real deployment and support fair model comparisons. Use when defining prediction timing, feature eligibility, train-validation-test splits, baselines, metrics, or controlled model iterations; not for auditing whether raw labels are trustworthy.

日本語の概要は準備中です。原文の説明を表示しています。

aiopshwang/data-analysis-ml-agent-skills122026年8月27日 更新

Orchestrate an end-to-end data analysis or machine learning project from decision framing through reproducible handoff. Use when a request spans multiple lifecycle stages or an ambiguous modeling request must become a decision-ready result; use narrower audit or experiment-design skills for isolated reviews.

日本語の概要は準備中です。原文の説明を表示しています。

aiopshwang/data-analysis-ml-agent-skills122026年8月27日 更新

Package completed data analysis and ML work so an independent recipient can reproduce the claimed results, verify artifact lineage, and operate the handoff within its stated scope. Use when finalizing a project, study, model package, or review bundle; not for deploying to a live system.

日本語の概要は準備中です。原文の説明を表示しています。

aiopshwang/data-analysis-ml-agent-skills122026年8月27日 更新

Route data analysis and machine learning work to the right skill in this suite. Use when starting any analysis, modeling, validation, or reproducibility task and the matching specialized skill is not yet clear; not needed when one specific skill already clearly applies.

日本語の概要は準備中です。原文の説明を表示しています。

aiopshwang/data-analysis-ml-agent-skills122026年8月27日 更新

Validate trained models and analytical claims against their intended decision, independent evidence, and human-reviewed ground truth. Use when reviewing model performance, analysis conclusions, launch claims, or evaluation reports; use failure diagnosis instead when the main task is locating a known defect.

日本語の概要は準備中です。原文の説明を表示しています。

aiopshwang/data-analysis-ml-agent-skills122026年8月27日 更新

aiopshwang のスキルをすべて見る

このスキルの問題を報告する