본문으로 건너뛰기
cccskills
무료GitHub에 공개

experiment-sdk-testing

Umbrella for experimentation-SDK test harnesses: the shared offline-datafile / hermetic-init pattern (commit a point-in-time flag/experiment config fixture, initialize the SDK with no network, pin arms per test, assert assignment integrity), with per-vendor references for Statsig (localMode + overrideGate), Optimizely (datafile + forced decisions), Split.io / Harness FME (localhost mode + features map or YAML fixture), Amplitude Experiment (local evaluation + bootstrap), and VWO (settings file + deterministic bucketing). Use when writing tests for application code instrumented with any of these five experimentation SDKs; for experiment DESIGN gates use ab-test-validity-checklist, and to read results use experiment-results-interpreter.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

testland/qa스타 13(GitHub 13 + cccskills 0)

실행 점수 미검증

설치 방법 보기

포함된 파일(9)

  • SKILL.md6.2 KB
  • .tessl-plugin/plugin.json882 B
  • references/amplitude.md5.4 KB
  • references/optimizely-recipes.md1.7 KB
  • references/optimizely.md5.7 KB
  • references/split-io-example.md1.4 KB
  • references/split-io.md10.8 KB
  • references/statsig.md4.9 KB
  • references/vwo.md4.6 KB

SKILL.md(원문)

설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.

experiment-sdk-testing

Overview

Every major experimentation SDK ships the same hermetic-test mechanism under a different name: a point-in-time config fixture (datafile, settings file, flag payload, features map) that the SDK evaluates locally, so tests make zero network calls, pollute no production analytics, and stay deterministic. The vendor-specific mechanics differ only in how the fixture is loaded and how an arm is pinned.

Routing table

SDKOffline mechanismArm pinningReference
StatsiglocalMode: trueoverrideGate / overrideConfigreferences/statsig.md
OptimizelyJSON datafile fixtureset_forced_decisionreferences/optimizely.md (+ optimizely-recipes.md)
Split.io / Harness FMEauthorizationKey: 'localhost' + features map / YAMLPer-key fixture entry (no override API)references/split-io.md (+ split-io-example.md)
Amplitude ExperimentLocalEvaluationClient + bootstrapFixture edit or evaluateV2 mockreferences/amplitude.md
VWOSettings file + is_development_modeDeterministic bucketing on user IDreferences/vwo.md

When to use

  • Tests for code that reads a gate / experiment / variant from any of the five SDKs above.
  • Assignment-integrity tests per ab-test-validity-checklist Step 3.
  • CI pipelines that must run without vendor network access.

The shared hermetic-init pattern

Regardless of vendor, the suite has the same five steps:

  1. Export and commit the config fixture - the datafile / settings / flag payload the SDK would fetch, checked into tests/fixtures/ and refreshed deliberately (drift between fixture and prod config is invisible otherwise).
  2. Initialize the SDK offline - the vendor's no-network switch (localMode, datafile string, 'localhost' key, bootstrap, is_development_mode).
  3. Pin the arm where the test needs one - override API, forced decision, per-key fixture entry, or a deterministically-bucketed user ID.
  4. Assert on values and keys, never internal IDs - variation keys and returned values survive environment changes; internal config IDs don't.
  5. Tear down - shutdown / destroy the client so event-flush timers and handles don't leak across test files.

Plus two integrity tests every suite should carry:

  • Determinism - the same user ID gets the same arm on repeated evaluation.
  • Distribution - across many user IDs, more than one arm actually occurs (and, where the split is known, roughly matches it).

Worked example (Optimizely datafile)

The team ships a new_checkout_flow flag with a treatment_a variation and needs a deterministic test that a premium-plan user is routed into the treatment:

import json
from optimizely import optimizely

# Step 1-2: committed fixture, offline init - no SDK key, no network
with open("tests/fixtures/optimizely-datafile.json") as f:
    client = optimizely.Optimizely(f.read())

def test_premium_user_in_treatment():
    # Step 3: context carries the attributes targeting needs
    user = client.create_user_context("user-1", {"plan": "premium"})
    decision = user.decide("new_checkout_flow")
    # Step 4: assert on enabled + variation_key, not IDs
    assert decision.enabled is True
    assert decision.variation_key == "treatment_a"

def test_assignment_deterministic():
    user = client.create_user_context("user-1")
    d1 = user.decide("new_checkout_flow")
    d2 = user.decide("new_checkout_flow")
    assert d1.variation_key == d2.variation_key

The fixture drives the whole decision; the same shape translates to each vendor via its reference above.

Anti-patterns (all vendors)

Anti-patternWhy it failsFix
Live API key in testsProduction analytics polluted; rate limits; flakesThe vendor's offline switch
Fixture not version-controlledTests flake when prod config changesCommit; refresh deliberately
Overrides / forced decisions leak across testsCross-test pollutionPer-test context + cleanup
Asserting on internal config / variation IDsIDs change per environmentAssert keys and values
Skipping client shutdown / destroyEvent-flush timers and handles leakTeardown in afterAll
Trusting one user ID to cover both armsMay bucket into one arm onlyDistribution test across many IDs

Limitations

  • Fixtures are point-in-time. Drift against the vendor UI is invisible until refreshed.
  • Offline modes don't validate the vendor's server-side analysis. Platform statistics are the vendor's job; these tests cover your code's SDK interaction only.
  • Per-vendor gaps (Statsig localMode still pings on some init paths; Split.io has no override API; Amplitude local evaluation lacks some flag types; VWO has no forced-decision API) are documented in each reference.

References

리뷰

아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.

댓글

로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.

아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.

보내면 바로 공개됩니다. 이메일 주소나 전화번호 같은 개인 정보는 쓰지 마세요. URL은 2개까지입니다.

0 / 1,000자

이 양식은 reCAPTCHA로 보호되며 Google 개인정보처리방침 및 서비스 약관이 적용됩니다.

같은 저장소의 스킬

a11y-violation-gate

무료

Builds a CI gate that fails the build on **new** WCAG / a11y violations introduced by a PR while grandfathering pre-existing violations on a per-rule / per-page baseline. Aggregates verdicts from axe-core / pa11y / Lighthouse a11y / WAVE / IBM Equal Access scans. Use when a project has accumulated a11y debt and a strict "zero violations" gate would block every PR - the ratchet pattern lets the team ship while preventing regressions.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

testland/qa132026년 10월 20일 업데이트

ab-test-validity-checklist

무료

Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

testland/qa132026년 10월 20일 업데이트

add-to-homescreen-flow-tests

무료

The single install-flow skill: the reference contract for the PWA install flow (installability gate fields, `beforeinstallprompt` handshake, per-platform paths - Android WebAPK / iOS Share menu / Firefox no-op - and the `display-mode` post-install signal, in references/install-flow-reference.md) plus the build-an-X workflow that emits the Add-to-Home-Screen suite. Walks the four-stage timeline, emitting one test per gate cell per [web.dev/articles/install-criteria][install-criteria], the deferred-prompt → `prompt()` → `userChoice` chain per [web.dev/articles/customize-install][customize-install], the iOS Safari manual-metadata branch (`apple-touch-icon`) per [web.dev/learn/pwa/installation][learn-pwa], and the post-install `(display-mode: standalone)` MQ assertion. Output: a Playwright spec with per-stage cells plus a coverage matrix. Use when a PWA's manifest, icons, or install handler change, when install conversion drops at an unknown stage, or when triaging a flaky install assertion.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

testland/qa132026년 10월 20일 업데이트

ai-spec-coverage-mapper

무료

Build-an-X workflow that uses an LLM to map existing tests to spec sections - given a spec doc + the test suite, the LLM identifies which tests cover which sections, surfaces uncovered sections (gap), and recommends specific tests to add. Output is a coverage matrix per spec ID. Scope is mapping tests that already exist and naming the gaps, not authoring tests for new acceptance criteria. Use when a spec doc and a test suite both exist but nobody can say which requirements are actually covered - before a release sign-off, an audit, or a decision about where to spend the next round of test effort.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

testland/qa132026년 10월 20일 업데이트

ai-test-generator

무료

Generates tests from natural-language specs (acceptance criteria, user stories, requirements) using an LLM, with confidence scoring per test case (LLM self-assessment plus heuristics: assertion quality, naming, completeness), batching uncertain cases for human review, and integration with the team's existing test framework. Use when the user asks to generate unit tests from acceptance criteria, convert user stories to test cases, automate test creation from requirements, or augment a spec-driven test suite with AI-generated stubs that are then curated before merge.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

testland/qa132026년 10월 20일 업데이트

allure-reports

무료

Configures Allure Report (test-runner adapter install, `allure-results` directory wiring, `categories.json` for failure classification, `history-trend.json` retention via the copy-history-between-runs pattern), runs the Allure CLI to convert `allure-results` to a static HTML site, and uploads the report as a CI artifact. Use when the team needs richer test reporting than JUnit XML - step-level attachments, per-test history, retry tracking, and severity / epic / feature labeling across framework-agnostic adapters (pytest, Jest, JUnit, TestNG, NUnit, Mocha). As a rich static HTML report generator, it is the open-source alternative to the sunset ExtentReports (JVM/.NET per-test HTML narrative); for hosted cross-run flakiness analytics rather than a static per-run report use currents-integration.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

testland/qa132026년 10월 20일 업데이트

testland의 스킬 모두 보기

이 스킬의 문제 신고