Testland

Compute group fairness metrics (selection rate, demographic parity, equalized odds) per sensitive feature with `MetricFrame`, then mitigate disparities using Reductions algorithms (`ExponentiatedGradient` with constraint = `DemographicParity`/`EqualizedOdds`). Wire group-disaggregated assertions into the model-evaluation gate. Use when a model's decisions affect people and a stakeholder, auditor, or regulation (ECOA, GDPR Art. 22, EU AI Act high-risk) requires evidence of per-group outcomes, or when someone reports the model treats a specific group worse.

The English summary is being prepared. Showing the original description.

Use Evidently OSS (100+ evaluation metrics, declarative testing API) to detect data drift, target drift, and model-performance regression, wired into CI as a gate (a Report run with include_tests) and into production monitoring as a continuous check; reports as HTML + JSON for both human review and pipeline assertions. Includes a drift-alert triage playbook: classify the fired alert's signal, rank root-cause hypotheses (upstream schema change, pipeline bug, training-serving skew, seasonality, genuine population shift), and pick rollback, retrain, quarantine, or alert re-tuning. Use when you need a drift or quality gate, a scheduled monitoring job, or a structured triage of a fired drift alert, for a tabular ML model. Built on the Evidently API specifically: for DeepChecks-based validation suites use deepchecks-tests instead.

The English summary is being prepared. Showing the original description.

Run Deepchecks suites (data integrity, train-test validation, model evaluation) on tabular / NLP / vision data + models. Pass `result.passed_conditions()` to CI to gate on regressions; the same checks run during research, CI, and production monitoring per the Deepchecks lifecycle posture. Use before training to catch train-test leakage and data-integrity defects in a tabular, NLP, or vision dataset, and to re-run the same suite on production samples to detect drift.

The English summary is being prepared. Showing the original description.

Derives human-readable manual test cases from stateful behavior: identify states, events, transitions, and guard conditions, draw the state table including invalid (empty-cell) transitions, choose a coverage level (all states, valid transitions / 0-switch, transition pairs / 1-switch per Chow, all transitions including invalid ones), then derive one test case per coverage item as an event sequence with per-step expected states (ISTQB CTFL v4.0 section 4.2.4). A deep single-technique walkthrough rather than a broad multi-lens case matrix; the output is manual step/expected cases rather than parameterized test code, and it covers how cases are derived rather than how a case record is structured. Use for lifecycle entities (accounts, orders, subscriptions), workflows, and UI wizards where the response to an event depends on the current state.

The English summary is being prepared. Showing the original description.

Builds stakeholder-readable scripted manual test cases from a feature spec in four formats: a step-table (preconditions / steps / expected result / actual / pass-fail / notes) for spreadsheet review, a Gherkin Given/When/Then format for BDD-aware teams, a business-language UAT script with acceptance-criteria mapping and contractual sign-off (references/uat-format.md), and a one-line-per-item execution checklist for smoke / on-call / bug-bash / compliance sweeps (references/checklist-format.md). Each script is self-contained (no implicit team knowledge), single-scenario (one happy + N edge per script), and includes the data setup the tester needs without being a developer. Use when a feature can't be (or shouldn't be) fully automated and a human tester needs an executable script or checklist - UAT sign-off rounds, regression baselines, certification testing, deploy smoke checklists, exploratory follow-up scripts.

The English summary is being prepared. Showing the original description.

Plans and runs time-boxed exploratory testing when tester hours are scarce before a release - one tester with two free 45-minute blocks before code freeze, a high-stakes window such as year-end payroll, or a device and environment the scripted suite never touches. Session-based per the Bachs' SBTM: charters (Explore X with Y to discover Z), 60-90 minute sessions, session sheets with TBS metrics, and the PROOF debrief. Bundles the exploration heuristics as references - Whittaker's seven tours, Kelly's FCC CUTS VIDS, Bach's SFDPOT. Broader than exploratory-charter-author, which writes one charter document: this owns the whole cycle from budgeting the available hours to debriefing what was found. Use when deciding what to explore with the time available, and how to run and record those sessions.

The English summary is being prepared. Showing the original description.

Configures policy-as-code testing using OPA / Conftest / Cedar - authors policies in Rego (OPA's language), runs Conftest against Kubernetes manifests / Terraform plans / Dockerfiles / arbitrary structured data, integrates with CI for PR-time policy gates. Per OPA's docs: "an open source, general-purpose policy engine that unifies policy enforcement across the stack." Use to express + enforce custom policies (cost limits, tagging requirements, security baselines) that Checkov / tfsec / KICS don't cover.

The English summary is being prepared. Showing the original description.

Author and run QEMU system emulation as an embedded-test target - qemu-system-arm / qemu-system-aarch64 / qemu-system-riscv32 launching cross-compiled ELF binaries on virtual MCUs and SoCs. Covers machine selection (-M virt / mps2-an385 / mps2-an386 / mps2-an500 / mps2-an511 / mps3-an524 / lm3s6965evb / raspi3b / xilinx-zynq-a9), CPU selection (-cpu cortex-m0 / cortex-m3 / cortex-m4 / cortex-m33 / cortex-a15 / cortex-a57 / max), -kernel ELF load, -nographic + -serial stdio, ARM semihosting via -semihosting-config enable=on,target=native (so cross-compiled GoogleTest / Unity binaries print to host stdio and exit with the test return code), GDB stub via -S -gdb tcp::1234, QMP monitor via -qmp tcp:host:port for automated test orchestration, and CI wiring. Use when host-only test runs are insufficient and the team wants arch-correct (endianness / alignment / interrupt-vector) behaviour on a virtual MCU without committing to physical hardware-in-loop.

The English summary is being prepared. Showing the original description.

Author and run ThrowTheSwitch Unity (the C unit-testing library) for bare-metal and RTOS C code. Distinct from the Unity game-engine Test Framework at docs.unity3d.com: this is the ThrowTheSwitch C testing library at throwtheswitch.org/unity, a single C file plus headers that runs on 8-bit MCUs through 64-bit hosts. Anchored on the Unity assertion API and configuration macros regardless of execution environment: the TEST_ASSERT_EQUAL_* / _FLOAT / _DOUBLE / _STRING / _MEMORY / _BITS assertion families, setUp/tearDown/RUN_TEST/UNITY_BEGIN/UNITY_END semantics and the exit-code contract, the generate_test_runner.rb generator, build-time config defines (UNITY_INCLUDE_DOUBLE, UNITY_OUTPUT_CHAR, UNITY_EXCLUDE_SETJMP), and CI integration via Ceedling JUnit XML; applies to host builds, cross-builds, and QEMU-run targets alike. For QEMU machine flags, semihosting, and exit-code capture, see qemu-system-test-runner. Use when the unit-under-test is pure C and the target ranges from 8-bit AVR to Cortex-M0 to Linux ARM.

The English summary is being prepared. Showing the original description.

Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.

The English summary is being prepared. Showing the original description.

Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision - with the deep methodology in references/: the peeking problem and its corrections (fixed-horizon, alpha-spending, always-valid mSPRT) in references/peeking.md, and guardrail-metric methodology (taxonomy, OEC relationship, pre-commitment, thresholds) in references/guardrails.md. Use when a data scientist or PM is ready to draw conclusions from an experiment, when designing a stop-early policy, or when declaring an experiment's guardrail set. Distinct from ab-test-validity-checklist (harness setup and SRM detection).

The English summary is being prepared. Showing the original description.

Umbrella for experimentation-SDK test harnesses: the shared offline-datafile / hermetic-init pattern (commit a point-in-time flag/experiment config fixture, initialize the SDK with no network, pin arms per test, assert assignment integrity), with per-vendor references for Statsig (localMode + overrideGate), Optimizely (datafile + forced decisions), Split.io / Harness FME (localhost mode + features map or YAML fixture), Amplitude Experiment (local evaluation + bootstrap), and VWO (settings file + deterministic bucketing). Use when writing tests for application code instrumented with any of these five experimentation SDKs; for experiment DESIGN gates use ab-test-validity-checklist, and to read results use experiment-results-interpreter.

The English summary is being prepared. Showing the original description.

Feature-flag test matrix design: the flag-state combinatorics problem (N flags × M variants × K user-segments = N×M×K test cases), the canonical coverage strategies (pairwise interaction coverage; default-only smoke; full matrix; risk-driven matrix), the workflow for building the coverage suite from a flag inventory (grep-based inventory, per-flag classification, PICT pairwise generation, per-cell test skeletons), the dedicated kill-switch test categories (references/killswitch.md: graceful degradation, fail-static default, kill latency, mid-flight consistency), and the flags-vs-experiments distinction. Use when designing the flag-test surface for a new project, building or auditing flag-test coverage, or authoring kill-switch tests.

The English summary is being prepared. Showing the original description.

Wraps LaunchDarkly server-side SDK testing patterns: TestData data source for hermetic tests (no network), file-based data source for fixture-driven tests, flag override patterns (TestData.update for per-test flag values), and assignment-integrity tests. Use when writing tests for code that uses LaunchDarkly flags; to decide which flag combinations those tests should cover in the first place, see feature-flag-test-matrix-reference.

The English summary is being prepared. Showing the original description.

Wraps OpenFeature (CNCF vendor-neutral SDK abstraction) testing patterns: the InMemoryProvider for hermetic tests without network calls, provider registration via OpenFeature.setProvider, the getBooleanValue/getBooleanDetails evaluation API with EvaluationDetails (value, variant, reason, errorCode), hooks for evaluation side-effects, and evaluation context for targeting-rule tests. Covers TypeScript, Java, and Python SDKs, plus per-vendor hermetic-bootstrap references for Unleash (bootstrap toggles), Flagsmith (offline LocalFileHandler), and GrowthBook (initSync payload). Use when writing tests for code that resolves feature flags through the OpenFeature SDK or the Unleash / Flagsmith / GrowthBook native SDKs; LaunchDarkly has its own skill (launchdarkly-testing).

The English summary is being prepared. Showing the original description.

Locates the condition a known-flaky test actually depends on by holding the test constant and varying one axis at a time (isolation, execution order, worker count, viewport, network latency, repetition depth), recording a pass/fail count per variation, and testing whether the gap between two conditions exceeds sampling noise. Covers choosing the run count N from the failure rate you need to detect, binomial confidence intervals on a measured reproduction rate, a two-proportion comparison rule, what a zero-failure result does and does not prove, and the resource-collision walk (DB row, DB schema, file path, port, env var, module state, inode, cookie jar) used once parallelism is implicated. Use when a specific test is already known to fail intermittently, reading its source has not explained why, and a decision about what to change must rest on measurement rather than on a plausible-sounding guess.

The English summary is being prepared. Showing the original description.

Builds a persistent flakiness infrastructure dashboard from JUnit XML or JSON CI run history: defines the flake-rate metric (failures per test over a configurable window), authors the data model, generates a Grafana time-series panel JSON or configures a Datadog CI Visibility view, derives the quarantine-candidate query, and wires trend alerts. Also generates the periodic (weekly / monthly) test-suite trend report - total runs, suite duration, flakiness rate, top failing tests, time-to-green per PR, week-over-week deltas - as a markdown summary for a team Slack channel or wiki page. Use when a team needs a long-lived observability surface for test reliability, or a scheduled comparable health report on top of it.

The English summary is being prepared. Showing the original description.

Reference catalog of the eight flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics, remediation per pattern, and the concrete code-level fixes: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, per-worker DB schemas via workerIndex, try/finally teardown, mocking network + clock at the boundary, stable role-based locators, TZ pinning, and RNG seeding. Use when triaging an unknown flake to identify the category before bisecting, or when a classified flake needs the specific code change to apply.

The English summary is being prepared. Showing the original description.

Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.

The English summary is being prepared. Showing the original description.

Coverage-guided fuzzing across every mainstream engine - libFuzzer (C/C++ in-process), AFL++ (out-of-process, QEMU mode for closed-source binaries), cargo-fuzz (Rust), Go native fuzzing (go test -fuzz), Atheris (Python), and Jazzer (JVM, @FuzzTest). Body covers choosing the right fuzzer for the language and build type (the routing tree) plus the engine-generic workflow: writing a small deterministic fuzz target, seed-corpus + dictionary construction, sanitizer selection (ASan + UBSan default, compatibility matrix), corpus minimisation, crash-artifact handling, and CI smoke-fuzz wiring with a cached corpus. Per-engine depth (flags, harness syntax, CI jobs) lives in references, as do the corpus-management and sanitizer-integration catalogs. Use when a project needs fuzz coverage and no fuzzer is chosen yet, or when authoring / running / maintaining a fuzz campaign with any of these engines. For triaging the resulting crashes see crash-triage-reference.

The English summary is being prepared. Showing the original description.

Reference catalog for triaging fuzzer crash artifacts - reading ASan, UBSan, and MSan output; classifying findings as LIKELY-EXPLOITABLE, MEDIUM, or BENIGN; deduplicating by stack-hash; minimizing reproducers with -minimize_crash; plus the bulk triage workflow for a full artifact directory (inventory, reproduce, classify, dedupe, BLOCK/PASS verdict with refuse-to-proceed rules and a report template). Use when you need to understand what a specific crash means, build exploitability intuition, work a set of findings by hand, or run a campaign-level triage that ends in a release verdict.

The English summary is being prepared. Showing the original description.

Profiles game builds against frame-time, memory, GPU draw-call, and GC-spike budgets using Unity Profiler + Profile Analyzer + Performance Testing package and Unreal Insights + stat commands. Establishes pass/fail thresholds (16.6 ms at 60 fps, 33.3 ms at 30 fps), writes automated performance regression tests that run in CI, and emits a structured budget report per SKU. Use when a title must hit a declared frame-time or memory budget before a milestone gate or platform-cert submission, or when a recent change needs a performance regression check.

The English summary is being prepared. Showing the original description.

Build a deterministic gameplay record/replay test artefact for Unity, Unreal, or Godot - record a player session, save it to disk, replay it bit-for-bit, and assert that the resulting game state matches the original. Covers Unity Input System's InputEventTrace API (Enable / Disable / WriteTo / ReadFrom / Replay) for input-level capture, Unreal's Replay System (DemoRec / DemoPlay / DemoStop console commands plus DemoNetDriver + NetworkReplayStreamer, default storage at %LOCALAPPDATA%/{Project}/Saved/Demos) for replication-stream capture, and Godot's community-pattern deterministic-RNG + input-script replay since Godot ships no first-party replay system. Use when authoring a regression-test artefact for player-recorded sessions, building a netcode replay for spectator / esports, or producing reproducible bug repros for cert teams.

The English summary is being prepared. Showing the original description.

Author and run GUT (Godot Unit Test) - the community-canonical GDScript test framework at github.com/bitwes/Gut and gut.readthedocs.io. Covers install (Godot Asset Library or manual `addons/gut/` copy + plugin enable), GUT panel inside the editor, writing tests that extend GutTest with `test_` prefix methods, the assertion family (assert_eq / assert_almost_eq / assert_true / assert_signal_emitted), lifecycle hooks (before_each / after_each / before_all / after_all), inner classes for grouping, parameterized tests via `params=[...]`, doubles / stubs / spies, async / coroutine tests, the command-line runner (`-d -s addons/gut/gut_cmdln.gd -gdir=res://test -gjunit_xml_file=... -gexit`), JUnit XML export, and CI integration. Godot 4.x uses GUT 9.x (current main branch supports 4.6.x; godot_4_7 branch for 4.7.x); Godot 3.x uses GUT 7.x. Use when the unit under test is GDScript code in a Godot project.

The English summary is being prepared. Showing the original description.