Results for “experiment”

A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked. Hypothesis writing, sample size, duration, segment analysis, running discipline, matching a result to a pre-committed decision rule, and the common failure modes that produce confidently wrong shipping decisions. Use this skill whenever the user is planning a test that has not run yet: framing a hypothesis, sizing the sample, setting duration, choosing guardrails, or deciding whether something is worth testing at all. Triggers on design an experiment, experiment plan, A/B test, split test, multivariate test, holdout, experiment hypothesis, sample size, minimum detectable effect, MDE, test duration, guardrail metric, no peeking, pre-committed decision rule, is this worth testing. Use `experimentation-analytics` instead when the test has already run and the question is how to read the result panel.

The English summary is being prepared. Showing the original description.

The science of learning through controlled experimentation. A/B testing isn't about picking winners—it's about building a culture of validated learning and reducing the cost of being wrong. This skill covers experiment design, statistical rigor, feature flagging, analysis, and building experimentation into product development. The best experimenters know that every test, positive or negative, teaches something valuable. Use when "a/b test, experiment, hypothesis, statistical significance, sample size, feature flag, variant, control, treatment, p-value, conversion rate, test winner, split test, experimentation, testing, statistics, feature-flags, hypothesis, growth, optimization, learning, validation" mentioned.

The English summary is being prepared. Showing the original description.

Design and document a marketing growth experiment. Use when the user says "run a growth experiment", "test this channel", "growth test", "I want to experiment with [channel]", "traction experiment", "acquisition experiment", "let's test if [channel] works for us", "bullseye framework", "which channels should we try", "run a traction test", or wants to systematically test a marketing or acquisition channel before committing budget. For page-level conversion hypotheses, see page-cro. For copy variants to test, see ab-copy-writer. For recording the result, see retro.

The English summary is being prepared. Showing the original description.

Focused Signals scout for PostHog projects running A/B experiments. Watches running experiments for validity threats (sample ratio mismatch, multi-variant contamination, exposure stalls, mid-run flag mutations) and lifecycle drift (zombie experiments running long past their useful life, decided-but-still-running experiments, ended experiments whose flags still serve multiple variants). Emits findings only when they clear the confidence bar; otherwise writes durable memory and closes out empty. Self-contained peer in the signals-scout-* fleet — no dependencies on other skills.

The English summary is being prepared. Showing the original description.

Write a rigorous, decision-grade experiment spec — falsifiable hypothesis, primary metric, guardrails, randomisation unit, exposure definition, method (A/B, holdout, switchback, quasi-experiment, MAB), MDE/duration plan, peek policy, validity threats, and pre-committed decision rule. Platform-agnostic. Load when the user has a candidate experiment and needs to spec it before launch, or says "spec this experiment", "write the test plan", "design this A/B test", "what's the hypothesis", "how big a sample do we need", "how long should we run this", "define the metrics for this test", or when the experimentation orchestrator routes here.

The English summary is being prepared. Showing the original description.

The science of learning through controlled experimentation. A/B testing isn't about picking winners—it's about building a culture of validated learning and reducing the cost of being wrong. This skill covers experiment design, statistical rigor, feature flagging, analysis, and building experimentation into product development. The best experimenters know that every test, positive or negative, teaches something valuable. Use when "a/b test, experiment, hypothesis, statistical significance, sample size, feature flag, variant, control, treatment, p-value, conversion rate, test winner, split test, experimentation, testing, statistics, feature-flags, hypothesis, growth, optimization, learning, validation" mentioned.

The English summary is being prepared. Showing the original description.

Organize repeated experiments to improve models, prompts, or agent systems against measurable goals. The skill records hypotheses and lessons in a persistent tree, isolates trials, and compares promising candidates using evaluation data kept separate from development.

• Compare repeated experiments to improve model training
• Improve an agent system's measured success rate
• Test prompt changes against a fixed evaluator

Use MLflow to record machine learning experiments, compare results, and manage model versions. The skill covers logging training settings, metrics, and files, organizing a model registry, and preparing models for local or cloud deployment.

• Comparing training settings and model performance
• Recording experiments for later reproduction
• Managing model versions and release stages

Use W&B to record machine learning experiments, visualize training, and compare results. The skill also covers automated searches for training settings and version tracking for datasets and models, helping teams share experiments and trace their inputs.

• Logging training metrics and viewing progress in real time
• Comparing experiments with different training settings
• Searching automatically for better training settings

Plan and analyze A/B tests, prepare tabular data, and train and evaluate prediction models. Estimate intervention effects from observational data, track model experiments with MLflow, and translate statistical findings into business decisions.

• Planning A/B test sample sizes and analyzing results
• Preparing tabular data for prediction models
• Comparing model performance and recording experiments

Plan low-effort experiments to check assumptions about an existing product before building a feature. Compare prototypes, A/B tests, and other methods, with a clear assumption, procedure, metric, and success threshold for each experiment.

• Checking feature assumptions before implementation
• Planning prototype usability tests
• Defining metrics and success thresholds for A/B tests

Evaluate whether completed experiments support a research claim using a separate Codex reviewer. Identify unsupported conclusions and missing evidence, then determine whether to revise the claim, run additional experiments, or move toward paper writing.

• Check research claims before drafting a paper
• Plan additional experiments for incomplete evidence
• Reconsider an approach after unsupported results

Check running experiments, collect logs and result metrics, and summarize progress alongside baseline comparisons. It supports SSH servers, Vast.ai, and Modal, with optional Weights & Biases metrics and cost reporting for the supported compute services.

• Check whether remote experiments have finished
• Compare collected metrics with the matching baseline
• Investigate NaN values or diverging training runs

Plan ablation studies, experiments that remove or replace parts of a machine learning method to test their contribution. Define reviewer questions, expected outcomes, priorities, execution order, and compute estimates, then implement experiments and record findings.

• Testing the contribution of individual method components
• Planning additional experiments before paper submission
• Prioritizing experiments within a GPU budget

Audit research experiments before using their results to support claims. An independent model checks reference data, metric calculations, result files, and evaluation scope, producing reports with evidence, findings, and specific actions to address concerns.

• Verify the source of reference data
• Check score normalization and metric calculations
• Match paper claims against saved results

Set up W&B to record machine learning metrics and settings, visualize training progress, and compare experiments. Includes automated searches for training settings, version tracking for datasets and models, and shared reports for team collaboration.

• Recording and visualizing training metrics
• Comparing experiments with different training settings
• Automatically searching for better training settings

Coordinate AI research from literature review and hypothesis testing to findings and paper drafts. The skill alternates focused experiments with broader reflection, maintains project records, and directs specialized skills to carry out research tasks.

• Turning a research question into testable hypotheses
• Managing repeated experiments across multiple hypotheses
• Resuming research from recorded findings and project state

Coordinate AI research from literature review to paper drafting. The skill alternates focused experiments with broader reviews of findings, tracks hypotheses and decisions, and uses specialist skills to carry out research tasks and prepare progress reports.

• Starting a research project from a question
• Testing and comparing multiple hypotheses
• Reassessing direction when experiments stall

Machine learning experimentation reference for model-experimentation conventions, experiment tracking and reproducibility, dataset and model abstractions, ML engagement fundamentals, and model-production readiness. Use when standing up ML experimentation infrastructure or assessing whether a trained model is ready for production.

The English summary is being prepared. Showing the original description.

Structures biological experiments so inference is valid by construction, covering Fisher's principles (randomization, replication, local control), the experimental-vs-observational unit distinction and pseudoreplication (Hurlbert 1984; Lazic 2018), randomization mechanics (complete, restricted, stratified, rerandomization, run-order), blocking layouts (randomized complete block, Latin square, incomplete block), factorial designs and interactions, and the split-plot/nested error strata hidden inside multi-batch genomics. Use when deciding the experimental unit and what counts as a replicate, planning randomization and run order, choosing a blocked/factorial/split-plot/nested layout, avoiding pseudoreplication in cell-culture or animal studies, or specifying the random-effects structure of the analysis model. For assigning samples to sequencing batches/lanes/plates and batch-effect correction see experimental-design/batch-design; for regulated clinical-trial randomization see clinical-biostatistics.

The English summary is being prepared. Showing the original description.

Pre-pipeline aggregator that scans AI agent cache directories (.claude, .cursor, .antigravity, .openclaw) or any user-specified directory for experimentation logs, extracts insights and numeric results, and formats them as PaperOrchestra-ready inputs (idea.md + experimental_log.md). TRIGGER when the user says "aggregate my agent logs for paper writing", "extract experiments from my coding agent history", "prepare PaperOrchestra inputs from my cache", "turn my agent logs into a paper", mentions a folder or directory they want to use as the basis for a paper, or wants to run PaperOrchestra but only has scattered agent experiment histories rather than structured inputs. Run this BEFORE paper-orchestra. Also called automatically by paper-orchestra when workspace/inputs/idea.md or workspace/inputs/experimental_log.md are missing.

The English summary is being prepared. Showing the original description.

Systematic experimental results analysis workflow for ML/AI research papers. Connects experimental data to publication-ready Results sections with statistical validation, visualizations, and quality checks. Triggers on "analyze experimental results", "generate results section", "statistical analysis of experiments", "compare model performance", "create results visualization".

The English summary is being prepared. Showing the original description.

Use when the user wants to design experiments, plan ablation studies, structure baselines, or create incremental evaluation strategies. Triggers on phrases like "design ablation", "plan experiment", "what experiments should I run", "baseline comparison", or "experiment matrix".

The English summary is being prepared. Showing the original description.

Audit existing experimentation infrastructure and past experiments for methodology issues. Use when asked to "audit our experiments", "is our experimentation sound", or "review past test methodology".

The English summary is being prepared. Showing the original description.