prod-canary-validator
Overview
A canary deploy sends the new version to a small slice of traffic,
watches metrics, and promotes only if they look healthy. "Looking
healthy" is usually a qualitative on-call judgment, so regressions
slip through. This skill builds the deterministic canary
verdict: a machine-checkable comparison of canary metrics vs
baseline that emits promote / pause / rollback, running the
analytical layer underneath the human review of the canary
observation step (canary-release).
When to use
- Canary deploys are part of the release process and the team wants
promote/rollback decisions backed by data, not just "looks fine."
- The team has metrics infrastructure (Datadog / Prometheus /
CloudWatch) capturing the right signals.
- A SLO depends on production verification; the canary is the
earliest gate where regressions can be caught.
Step 1 - Pick the metric set
Canary metrics should cover three classes:
| Class | Examples | Why |
|---|
| Reliability | Error rate (5xx %), failed-request count | The new code may crash. |
| Performance | p50 / p95 / p99 latency | The new code may be slow. |
| Business KPI | Checkout-completion rate, sign-up rate, revenue/min | The new code may break a flow without crashing. |
Pick 5-10 metrics. More = noisier verdict (more chances to
trip); fewer = blind spots.
Always include at least one business KPI - performance and
reliability metrics can both look fine while the actual user
outcome (checkout completion) regresses.
Step 2 - Set thresholds (absolute + relative)
Per metric, define two thresholds:
| Threshold type | Example |
|---|
| Absolute | "Error rate <0.5%" - a hard floor regardless of baseline. |
| Relative | "Error rate ≤ 1.5× baseline" - catches regressions even when baseline is high. |
# canary-thresholds.yml (excerpt)
metrics:
error_rate:
absolute: { max: 0.5 } # %
relative: { max: 1.5 } # 1.5× baseline
Full config for every metric:
references/canary-thresholds.yml.
The combination is essential: absolute catches "unacceptable
regardless"; relative catches "worse than the baseline by a
meaningful amount."
Step 3 - Statistical significance
A 1-minute window of canary data has high variance. The verdict
"canary error rate 0.4% vs baseline 0.3% - promote?" depends on
sample size:
- N=10 requests, 0.4% diff → meaningless; could be 1 vs 0
errors.
- N=10,000 requests, 0.4% diff → statistically significant;
worth flagging.
Use a two-sample test (proportion test for error rate,
Welch's t-test for latency) to compute a p-value, then gate each
metric: the absolute floor fails unconditionally, but the relative
limit fails only when the difference is statistically significant.
# core gate; full script: scripts/canary_verdict.py
def gate(metric, canary, baseline, t, p, alpha=0.05):
if 'max' in t.get('absolute', {}) and canary.value > t['absolute']['max']:
return 'fail-absolute'
ratio = canary.value / baseline.value
if 'max' in t.get('relative', {}) and ratio > t['relative']['max'] and p < alpha:
return 'fail-relative' # only when significant
return 'pass'
Full runnable implementation (compare_proportions,
compare_latencies, and the promote/pause/rollback classification):
scripts/canary_verdict.py.
alpha = 0.05 is convention; use 0.01 for high-criticality
metrics. Skip relative checks when not significant - otherwise
random variance triggers false rollbacks.
Step 4 - Observation window
Default: 30 minutes. Pattern:
| Window | Use |
|---|
| 5 min | Smoke check only - sanity; not the promote gate. |
| 15 min | Low-traffic services where 30 min wouldn't add sample size. |
| 30 min | Default for most services. |
| 1 hour | High-variance metrics (sparse business KPIs). |
| 2 hour | Pre-major-release; matches the team's release-engineering runbook. |
Per canary-release: the observation window is "early warning
for potential problems before impacting your entire production
infrastructure or user base." Longer = more signal, slower release.
Step 5 - Per-traffic-share scaling
Canary at 5% traffic with N requests/min collects 1/20 the sample
of baseline at 95% traffic. Adjust the statistical confidence
accordingly:
# Effective sample size correction
canary_share = 0.05 # 5%
baseline_share = 0.95
required_window = window_minutes * (1.0 / canary_share - 1.0)
# A 30-min observation at 5% canary needs the equivalent of 600 min
# at full traffic to match statistical power.
For very low canary shares (1%), prefer to bump the share before
verdict (5% canary for 30 min beats 1% canary for 2 hours on
sample-size grounds).
Step 6 - Output
## Canary verdict - `<release>` `<sha>`
**Window:** 30 minutes (14:00-14:30 UTC)
**Canary traffic share:** 5% (12,400 requests)
**Baseline:** main `def456` (235,800 requests)
**Verdict:** ⚠ PAUSE - investigate before promoting
### Per-metric
| Metric | Canary | Baseline | Δ | p-value | Verdict |
|----------------------------|-----------|-----------|----------|---------|---------|
| error_rate (%) | 0.42 | 0.31 | +35.5% | 0.018 | ⚠ relative threshold tripped |
| p95 latency (ms) | 245 | 240 | +2.1% | 0.62 | ✅ within threshold |
| p99 latency (ms) | 890 | 850 | +4.7% | 0.41 | ✅ within threshold |
| checkout_completion_rate (%) | 91.2 | 92.1 | -1.0% | 0.34 | ✅ within threshold |
| signup_rate (%) | 4.2 | 4.3 | -2.3% | 0.78 | ✅ within threshold |
### Recommendation
PAUSE. The error rate ratio (1.35x baseline) is statistically
significant (p=0.018) and exceeds the relative threshold (1.5x -
note: 1.35 < 1.5 absolute but the trend warrants investigation).
Investigate the new error categories before promoting.
### Investigation hand-off
- Top new error types in the canary window:
- `RateLimitExceeded` (12 occurrences; 0 in baseline) - possible
new dependency timeout.
- `NullPointerException at Cart.addItem:42` (3 occurrences; 0 in
baseline) - likely real regression.
Recommend: investigate the NPE before any promotion decision.
Step 7 - CI / orchestration integration
Wire as a step in the release pipeline:
- name: Promote to canary (5%)
run: ./deploy.sh --canary --share 5
- name: Wait for observation window
run: sleep 1800 # 30 min
- name: Compute verdict
id: verdict
run: |
python scripts/canary_verdict.py \
--canary-window "30m" \
--baseline-window "1h" \
--thresholds canary-thresholds.yml \
> verdict.json
echo "result=$(jq -r .verdict verdict.json)" >> "$GITHUB_OUTPUT"
- name: Promote to 100%
if: steps.verdict.outputs.result == 'promote'
run: ./deploy.sh --promote
- name: Pause for human review
if: steps.verdict.outputs.result == 'pause'
uses: trstringer/manual-approval@v1
with:
approvers: oncall-team
minimum-approvals: 1
- name: Rollback
if: steps.verdict.outputs.result == 'rollback'
run: ./deploy.sh --rollback
The verdict is the gate; the human reviews on pause (the
ambiguous case); rollback is automatic on clear failure.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
| Eyeballed verdict | Subjective; varies by who's on-call. | Deterministic verdict (Step 3-6). |
| Absolute thresholds only | Catches "always bad"; misses "regressed but still under absolute." | Both absolute + relative (Step 2). |
| Relative threshold without significance test | Random variance trips false rollback. | Skip relative when p > alpha (Step 3). |
| Single metric (error rate only) | Latency / business KPI regressions invisible. | 5-10 metrics across 3 classes (Step 1). |
| 5-min observation | Insufficient sample; high variance. | 30-min default (Step 4). |
| Auto-promote without human-review middle ground | Edge cases (1.4× baseline, p=0.06) get either stamped through or rolled back. | Three-state verdict (promote / pause / rollback) (Step 7). |
| Same threshold for every service | A 0.5% error rate may be normal for some services, alarming for others. | Per-service thresholds (Step 2). |
Limitations
- Sample size in low-traffic services. A canary at 1% of low
traffic produces no signal. Use traffic shadowing or
pre-production load instead.
- Noisy metrics. A flapping monitor can produce false
rollbacks; pair with metric smoothing (rolling N-min average).
- Business KPIs lag. "Checkout completion rate" needs minutes
to converge; longer windows for KPI-driven decisions.
- Regional differences. Per-region canary may have skewed
traffic; consider per-region verdicts.
- No semantic understanding. A 5% latency increase on the most
critical endpoint matters more than a 50% increase on a healthcheck;
weight metrics by business value (Step 1 metric class).
References
- cr - Martin Fowler on canary releases: gradual rollout,
monitoring + rollback, "early warning for potential problems
before impacting your entire production infrastructure or user
base."
synthetic-monitor-author - sibling: continuous-in-production verification (different
cadence, different goal).feature-flag-experiment-validator - sibling: A/B test analysis (different statistical framework
but related).