Fixing Gramax CI pipelines

Diagnose and repair red CI jobs on a branch of the Gramax product repo (ics/doc-reader, id 155). Sweep the branch's recent pipeline history — not just the newest pipeline — build a worklist of every distinct broken job, then fix them all in one run.

Runs headless in CI (see the ci skill for the shared environment) or locally from the team workspace. Non-interactive when headless: decide and act.

Harness run recipes + env + gotchas live in REFERENCE.md — read it before reproducing anything.

Target branch

• CI: $CI_COMMIT_BRANCH.
• Local: current checked-out gramax branch — git -C gramax rev-parse --abbrev-ref HEAD.

Algorithm

1. Sweep the pipeline history — build the worklist

Scan the last 20 pipelines on the branch ($SWEEP_DEPTH, default 20). One pipeline is not enough: a job can be red for weeks, be skipped by rules: in the newest run, or fail only intermittently. History is what tells those apart.

BRANCH=${CI_COMMIT_BRANCH:-$(git -C gramax rev-parse --abbrev-ref HEAD)}
DEPTH=${SWEEP_DEPTH:-20}

# 1. recent pipelines for the ref
glab api "projects/155/pipelines?ref=$BRANCH&per_page=$DEPTH" \
  | jq -r '.[]|"\(.id)\t\(.status)\t\(.updated_at)\t\(.sha[0:8])"'

# 2. every failed job across ALL of them — the raw sweep
for PID in $(glab api "projects/155/pipelines?ref=$BRANCH&per_page=$DEPTH" | jq -r '.[].id'); do
  glab api "projects/155/pipelines/$PID/jobs?per_page=100" \
    | jq -r --arg p "$PID" '.[]|select(.status=="failed")|"\($p)\t\(.id)\t\(.name)\t\(.created_at)"'
done | tee /tmp/failed-jobs.tsv

# 3. per job name: how many pipelines it failed in (frequency = flaky signal)
cut -f3 /tmp/failed-jobs.tsv | sort | uniq -c | sort -rn

# 4. decisive error for one occurrence — newest first
glab api "projects/155/jobs/$JOB_ID/trace" | tail -50

Group by job name, not by job id — the same broken next-e2e-pw across 8 pipelines is one worklist item, not eight. For each distinct job name record:

• Failure rate over the swept window (failed pipelines / pipelines that ran it).
• First bad pipeline — the oldest consecutive failure; the commit between it and the preceding green one is the suspect.
• Decisive error line, read from the newest occurrence.
• Root cause class (below). Different error signatures under the same job name = separate worklist items.

Classify each into one root cause:

• code bug — an app-source change broke the build or a test.
• test bug — the test is wrong/stale; product code is fine.
• runner infra — OOM, timeout, native-module rebuild failure, image/tag mismatch, external-dependency outage.
• flaky — fails intermittently across the window while the same commit also passed. History proves this; a single pipeline never can.

Then order the worklist: persistent breaks before flaky ones, oldest break first. Report the full worklist before starting — an item you skip must be named with a reason, never dropped silently.

2. Work the list — reproduce, fix, verify (per item)

For each worklist item, in order:

1. Reproduce. Follow the REFERENCE.md recipe for that harness — it documents the preflight (install-deps.sh, native better-sqlite3 rebuild, the gramax-core.node addon) and the exact run command. Confirm the same failure locally before touching code. Can't reproduce → mark the item unreproducible, report the env gap, move to the next item. Don't guess.
2. Fix. Apply the narrowest change for the cause class (targeted guard at the call site, not a global/config change).
3. Verify. Re-run that harness locally, confirm green. Not green → the item is unfixed; say so, don't count it as done.

Finish the whole list before landing anything. One hard item does not cancel the easy ones — a blocked item is reported, the rest still ship.

3. Land all fixes in ONE commit

All verified fixes from the run go into a single commit — not one per item.

Target branch	CI / test fixes only	Batch contains any app-code fix
release/*, develop	push direct	whole batch → one MR
any other branch	push direct	push direct

The batch is atomic: one app-code fix on release/*/develop sends the whole commit through an MR (title English, body Russian, ## Notes; use the merge-request skill). Never split the batch to route parts differently.

Commit message lists every fixed job by name, one line each, plus the items left unfixed and why.

• All writes bot-authored (GITLAB_CLAUDE_ACCESS_TOKEN) + initiator credit (.claude/scripts/gitlab-initiator → Co-Authored-By: <Full Name> <Email>).

4. Report

Final output = the worklist with an outcome per item: fixed / unreproducible / flaky, not patched / external, not ours. Nothing disappears from the list.

Cause → action

Cause	Action
code bug	fix source; land per table (MR on release/develop)
test bug	fix/adjust the test; it's a CI/test fix → push direct
flaky	history already shows whether it recurs — intermittent across the window and never reproducible locally → report it, don't patch product code on a guess
runner infra (ours)	fix CI config/resources (timeout, memory, native rebuild) → push direct
runner infra (external outage)	report, do not patch — nothing to fix in our code

Red flags — stop

• Looking at only the newest pipeline. Sweep the window — that's the point.
• Stopping after the first fixed item while the worklist still has entries.
• Dropping an item from the report because it was hard. Name it + say why.
• One commit per fix, or splitting a mixed batch across a push and an MR.
• Pushing a fix you never reproduced/verified locally.
• Patching product code for a flaky failure.
• A broad global/config change to silence one job — prefer the narrowest layer.
• Editing enterprise/ges harnesses — out of scope (see REFERENCE.md).