Results for “leakage”

Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".

The English summary is being prepared. Showing the original description.

Assessing privacy risks in large language model outputs including training data memorisation, PII leakage in generated text, prompt injection leading to data extraction, and hallucinated personal data. Covers output filtering, guardrails, and monitoring. Keywords: LLM privacy, output risk, memorisation, PII leakage, prompt injection, hallucinated PII.

The English summary is being prepared. Showing the original description.

敵対的分析手法を統合したレビューの entry skill。認知バイアス対策の3手法 （Pre-mortem / War Game / Logic Torturing）と、宣言・主張と実態の乖離を突く claim-vs-actual 検出3パターン（Self-Contradiction / Refactor-Claim Audit / Cross-File Leakage）へルーティングし、通常のレビューでは見えない設計の盲点・ 防御の穴・論理の弱点・宣言と実装のズレを可視化する。

The English summary is being prepared. Showing the original description.

Design evaluation tasks that check whether agents actually complete work. Define requests, simulated user behavior, starting conditions, and verifiable success criteria, then plan difficulty levels, training-data leakage checks, and ongoing maintenance.

• Creating evaluation tasks for real business workflows
• Reviewing success criteria and verification logic
• Designing checks for evaluation data leakage into training

Inspect supported local scientific files before modeling, summarize structure and missing values, and flag possible data leakage or outlier effects. Produces bounded aggregate reports with identifiers obscured and provides a structured analysis report template.

• Check local CSV or TSV data before modeling
• Audit missing values and overlap between data splits
• Compare outlier influence and transformation sensitivity

Design layered protections for AI agents against malicious instructions, tool misuse, and data leakage. Provides guidance and code examples for input checks, isolated tool execution, permission limits, and audit logging to help review and strengthen agent security.

• Review defenses against malicious instructions
• Isolate tool execution and restrict permissions
• Assess data leakage risks between tenants

Test whether an AI agent reveals sensitive information through progressively stronger dialogue prompts. Check API keys, personal data, system instructions, and internal settings, then report findings only when sensitive content is actually accessible.

• Check for API key and credential leaks
• Test exposure of system instructions
• Inspect personal data and internal settings

Review build and deployment pipelines for unsafe workflows, exposed credentials, and configuration flaws across GitHub Actions, Jenkins, GitLab CI, and Terraform state files. Use concrete evidence and false-positive checks to distinguish exploitable issues from suspicious settings.

• Review GitHub Actions workflows for unsafe execution paths
• Investigate Jenkins exposure and affected versions
• Check logs, artifacts, and container images for leaked secrets

Investigate Next.js security issues across Server Actions, middleware, image optimization, and cached pages. Provides testing workflows for authentication bypass and data exposure, with checks to distinguish actual vulnerabilities from normal responses.

• Checking authentication in Server Actions and middleware
• Testing image optimization for unintended server requests
• Inspecting page data for sensitive information

⚠️ CRITICAL USER EXPERIENCE-BASED SKILL - ALWAYS CONSULT BEFORE DATA PREPROCESSING ⚠️ Prevents catastrophic errors (88.9% error rate in V1.0 case study) through multi-level feature analysis, data leakage detection, and semantic validation. MANDATORY for: data preprocessing, feature engineering, standardization, normalization, interpolation, missing value handling, feature selection, or ANY data transformation task. Covers grouped time-series, cross-sectional, panel data. Detects: time travel leakage, causal inversion, ID misuse, semantic-numeric fallacies, distribution blindness. User's hard-won lessons from real project failures.

The English summary is being prepared. Showing the original description.

Detects and prevents data leakage in machine learning and mathematical modeling. Use after ML tasks involving data cleaning, feature engineering, data augmentation, algorithm development, normalization, missing value imputation, dimensionality reduction, feature selection, or time series modeling. Checks if features/statistics would be available at prediction time.

The English summary is being prepared. Showing the original description.

Use when you need framework-agnostic Hexagonal architecture guidance for Java projects - ports and adapters boundaries, application core independence, driving and driven adapters, dependency direction, infrastructure leakage detection, optional architecture tests, and evidence-backed remediation. This should trigger for requests such as Review Hexagonal architecture; Review ports and adapters boundaries; Enforce dependency direction; Add ArchUnit architecture tests; Detect infrastructure leakage in domain code; Improve ports and adapters boundaries. Part of Plinth Toolkit

The English summary is being prepared. Showing the original description.

Think and work like an expert Data Scientist. Use when a task calls for Data Scientist judgment. Reasons from CRISP-DM business estimands, leakage-safe sklearn Pipelines and nested CV, SQL/warehouse semantic metrics, A/B power and SRM/AA guardrails, causal DAG covariate discipline, and Model Cards/Datasheets while treating train-test leakage, Simpson's paradox, peeking, and PSI>0.25 drift as first-class failure modes.

The English summary is being prepared. Showing the original description.

Audit existing model training code — find reproducibility issues, data leakage, and missing best practices. Use when asked to "audit our training code", "is our training reproducible", or "check for training data leakage".

The English summary is being prepared. Showing the original description.

Audit feature engineering code for leakage, quality issues, and pipeline correctness. Use when asked to "audit our feature pipeline", "is there data leakage", or "find feature quality issues".

The English summary is being prepared. Showing the original description.

Audit existing eval coverage — gaps, metric validity, benchmark leakage, dataset freshness. Use when asked "what eval coverage do we have", "audit our evals", or "check for benchmark leakage".

The English summary is being prepared. Showing the original description.

Review a data analysis for quality, correctness, and reproducibility. Covers data quality assessment, assumption checking, model validation, data leakage detection, and reproducibility verification. Use when reviewing a colleague's analysis before publication, validating an ML pipeline before production deployment, auditing a report for regulatory or business decision-making, or performing a second-analyst review in a regulated environment.

The English summary is being prepared. Showing the original description.

Extract sensitive training data and internal capabilities from AI models through repeated programmatic querying. Use this skill when testing LLMs and ML models for data privacy leaks, intellectual property exposure, or unintended memorization. Covers membership inference, prompt extraction, training data reconstruction, and API boundary testing.

The English summary is being prepared. Showing the original description.

Checks layer leakage in method signatures, missing DTOs, entity leakage to API, inconsistent error contracts. Returns findings with penalty-based scoring.

The English summary is being prepared. Showing the original description.

Design leakage-safe machine learning experiments that mirror real deployment and support fair model comparisons. Use when defining prediction timing, feature eligibility, train-validation-test splits, baselines, metrics, or controlled model iterations; not for auditing whether raw labels are trustworthy.

The English summary is being prepared. Showing the original description.

モジュール/スキルの構造変更で当該ファイルは更新したが、caller 側が古い構造（旧参照・旧シグネチャ・旧採番）を参照したまま残るパターンを検出する

The English summary is being prepared. Showing the original description.

Tests APIs for excessive data exposure where endpoints return more data than the client application needs, relying on the frontend to filter sensitive fields. The tester intercepts API responses and analyzes them for leaked PII, internal identifiers, debug information, or sensitive business data that the UI does not display but the API transmits. This maps to OWASP API3:2023 Broken Object Property Level Authorization. Activates for requests involving API data leakage testing, excessive data e...

The English summary is being prepared. Showing the original description.

Validates predictive models on omics and biomedical data with nested cross-validation, group/batch/temporal-aware splits, the full data-leakage taxonomy, probability calibration, decision-curve net benefit, optimism correction, sample-size planning, and TRIPOD+AI reporting. Use when estimating model performance honestly, choosing a CV scheme, detecting leakage, or judging whether reported discrimination means the model is actually useful. For feature selection itself see machine-learning/biomarker-discovery; for confirmatory-trial inference see clinical-biostatistics/trial-reporting.

The English summary is being prepared. Showing the original description.

Tests APIs for excessive data exposure where endpoints return more data than the client application needs, relying on the frontend to filter sensitive fields. The tester intercepts API responses and analyzes them for leaked PII, internal identifiers, debug information, or sensitive business data that the UI does not display but the API transmits. This maps to OWASP API3:2023 Broken Object Property Level Authorization. Activates for requests involving API data leakage testing, excessive data exposure, response filtering bypass, or API over-fetching.

The English summary is being prepared. Showing the original description.