Manual E2E Test

Test a PR/branch end-to-end by building the full platform, interacting via browser and API, capturing screenshots, and reporting results.

Changelog 2.2.0 — auth flow updated for Better Auth (Supabase signup is gone), env-setup gaps closed, a proven Playwright fallback for agent-browser, a billing-test trap that produces false passes, safer process cleanup, and a mock-provider pattern for deterministic $0 testing. Learned on #14206 — see the evidence comment. 2.2.1 — corrects two claims from 2.2.0 that didn't survive live-stack verification (JWKS does not rotate on a frontend restart; the local Postgres port is 5432, not 54322) and hardens the auth setup (explicit password-length/allowlist/rate-limit failure modes, fail-fast on an empty token, password kept out of process args). 2.3.0 — screenshots are posted as GitHub comment attachments (gh pr comment --attach, needs gh >= 2.99.0) instead of being pushed to a test-screenshots/* branch, which accumulated one branch per tested PR and made every old report's images depend on that branch surviving.

Critical Requirements

These are NON-NEGOTIABLE. Every test run MUST satisfy ALL the following:

1. Screenshots at Every Step

• Take a screenshot at EVERY significant test step — not just at the end
• Every test scenario MUST have at least one BEFORE and one AFTER screenshot
• Name screenshots sequentially: {NN}-{action}-{state}.png (e.g., 01-credits-before.png, 02-credits-after.png)
• If a screenshot is missing for a scenario, the test is INCOMPLETE — go back and take it

2. Screenshots MUST Be Posted to PR

• Attach ALL screenshots to the PR comment with gh pr comment --attach (see Step 7). Never push image files to a repo branch — attachments live in GitHub's own asset store and leave nothing behind in the repo
• Post a PR comment with ALL screenshots embedded inline, each with its explanation
• This is NOT optional — every test run MUST end with a PR comment containing screenshots
• If the upload fails, retry. If it still fails, list failed files and require manual drag-and-drop/paste attachment in the PR comment, and mark the run INCOMPLETE

3. State Verification with Before/After Evidence

• For EVERY state-changing operation (API call, user action), capture the state BEFORE and AFTER
• Log the actual API response values (e.g., credits_before=100, credits_after=95)
• Screenshot MUST show the relevant UI state change
• Compare expected vs actual values explicitly — do not just eyeball it

4. Negative Test Cases Are Mandatory

• Test at least ONE negative case per feature (e.g., insufficient credits, invalid input, unauthorized access)
• Verify error messages are user-friendly and accurate
• Verify the system state did NOT change after a rejected operation

5. Test Report Must Include Full Evidence

Each test scenario in the report MUST have:

• Steps: What was done (exact commands or UI actions)
• Expected: What should happen
• Actual: What actually happened
• API Evidence: Before/after API response values for state-changing operations
• Screenshot Evidence: Before/after screenshots with explanations

State Manipulation for Realistic Testing

Billing-test trap — this one produces a false pass, not a visible failure. LLM block cost filters key on the platform-owned credential id. A run made with the test user's own API key bills nothing by design, so a credits before/after assertion silently passes on zero deltas either way. Any test that verifies credit reconciliation MUST use the system credential, not a user-supplied key. Related: Ollama block entries are configured with an explicit $0 run-based cost (BlockCostType.RUN, cost_amount=0 in block_cost_config.py), not a token-metered one — so pre/post-flight cost deltas are always 0 for them regardless of credential. Never use an Ollama model to test credit reconciliation; pick any hosted model billed through the system credential instead.

When testing features that depend on specific states (rate limits, credits, quotas):

1. Use Redis CLI to set counters directly:

   # Find the Redis container
   REDIS_CONTAINER=$(docker ps --format '{{.Names}}' | grep redis | head -1)
   # Set a key with expiry
   docker exec $REDIS_CONTAINER redis-cli SET key value EX ttl
   # Example: Set rate limit counter to near-limit
   docker exec $REDIS_CONTAINER redis-cli SET "rate_limit:user:$PR_TEST_USER_EMAIL" 99 EX 3600
   # Example: Check current value
   docker exec $REDIS_CONTAINER redis-cli GET "rate_limit:user:$PR_TEST_USER_EMAIL"
   

2. Use API calls to check before/after state:

   # BEFORE: Record current state
   BEFORE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits BEFORE: $BEFORE"
   
   # Perform the action...
   
   # AFTER: Record new state and compare
   AFTER=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits AFTER: $AFTER"
   echo "Delta: $(( BEFORE - AFTER ))"
   

3. Take screenshots BEFORE and AFTER state changes — the UI must reflect the backend state change

4. Never rely on mocked/injected browser state — always use real backend state. Do NOT use agent-browser eval to fake UI state. The backend must be the source of truth.

5. Use direct DB queries when needed:

   # Query via Supabase's PostgREST or docker exec into the DB
   docker exec supabase-db psql -U supabase_admin -d postgres -c "SELECT credits FROM user_credits WHERE user_id = '...';"
   

6. After every API test, verify the state change actually persisted:

   # Example: After a credits purchase, verify DB matches API
   API_CREDITS=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   DB_CREDITS=$(docker exec supabase-db psql -U supabase_admin -d postgres -t -c "SELECT credits FROM user_credits WHERE user_id = '...';" | tr -d ' ')
   [ "$API_CREDITS" = "$DB_CREDITS" ] && echo "CONSISTENT" || echo "MISMATCH: API=$API_CREDITS DB=$DB_CREDITS"
   

Arguments

• $ARGUMENTS — worktree path (e.g. $REPO_ROOT) or PR number
• If --fix flag is present, auto-fix bugs found and push fixes (like pr-address loop)

Step 0: Resolve the target

# If argument is a PR number, find its worktree
gh pr view {N} --json headRefName --jq '.headRefName'
# If argument is a path, use it directly

Determine:

• REPO_ROOT — the root repo directory: git -C "$WORKTREE_PATH" worktree list | head -1 | awk '{print $1}' (or git rev-parse --show-toplevel if not a worktree)
• WORKTREE_PATH — the worktree directory
• PLATFORM_DIR — $WORKTREE_PATH/autogpt_platform
• BACKEND_DIR — $PLATFORM_DIR/backend
• FRONTEND_DIR — $PLATFORM_DIR/frontend
• PR_NUMBER — the PR number (from gh pr list --head $(git branch --show-current))
• PR_TITLE — the PR title, slugified (e.g. "Add copilot permissions" → "add-copilot-permissions")
• RESULTS_DIR — $REPO_ROOT/test-results/PR-{PR_NUMBER}-{slugified-title}

Create the results directory:

PR_NUMBER=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json number --jq '.[0].number')
PR_TITLE=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json title --jq '.[0].title' | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | head -c 50)
RESULTS_DIR="$REPO_ROOT/test-results/PR-${PR_NUMBER}-${PR_TITLE}"
mkdir -p $RESULTS_DIR

Test user credentials — required to log into the UI or call authenticated APIs.

NEVER hardcode these in this SKILL, a PR comment, a screenshot, or any committed file. Sources, in priority order:

1. Env vars (CI / preconfigured local shell): $PR_TEST_USER_EMAIL + $PR_TEST_USER_PASSWORD. If both are set, use them.
2. Interactive prompt (everything else, including dev-preview runs): if the env vars are not set, ASK the user at the start of the run — e.g. "I need test-user credentials for this run; paste the email and password." Hold them in shell vars only for the duration of the run. Do not echo, log, or write them to disk.

Acquire the variables — env first, prompt if missing — and only then lock them in:

# 1. Prefer env vars (CI / preconfigured shell). Prompt only for the
#    specific var that is unset so an already-exported credential is
#    not overwritten by the prompt when only the other one is missing.
if [ -z "${PR_TEST_USER_EMAIL:-}" ] || [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
  echo "Test user credentials required for this run."
  if [ -z "${PR_TEST_USER_EMAIL:-}" ]; then
    read -r -p   "Email:    " PR_TEST_USER_EMAIL
  fi
  if [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
    read -r -s -p "Password: " PR_TEST_USER_PASSWORD
    echo
  fi
  export PR_TEST_USER_EMAIL PR_TEST_USER_PASSWORD
fi

# 2. Lock them in — fail loudly if either is STILL unset (e.g. the user
#    pressed Enter on an empty prompt). The error message names the var so
#    the agent / operator knows what to fix.
: "${PR_TEST_USER_EMAIL:?PR_TEST_USER_EMAIL is empty after env+prompt — supply a value before re-running}"
: "${PR_TEST_USER_PASSWORD:?PR_TEST_USER_PASSWORD is empty after env+prompt — supply a value before re-running}"

For local docker-compose runs, a fresh dev user is created on first call to the signup snippet below. For dev-preview runs, the test user lives in the project's hosted auth backend — ask the user for the current valid credentials each session (the previously-shared test@test.com test account was disabled on 2026-05-23 after its credentials leaked into this very SKILL — do NOT re-introduce a default). PR_TEST_USER_PASSWORD should always be a throwaway/test-only credential, never a real account's password — the auth requests in 3h go over plain HTTP on localhost:3000 for local runs, which has no transport encryption. If a dev-preview run's target isn't on localhost, confirm it's HTTPS before sending credentials to it.

Step 1: Understand the PR

Before testing, understand what changed:

cd $WORKTREE_PATH

# Read PR description to understand the WHY
gh pr view {N} --json body --jq '.body'

git log --oneline dev..HEAD | head -20
git diff dev --stat

Read the PR description (Why / What / How) and changed files to understand: 0. Why does this PR exist? What problem does it solve?

1. What feature/fix does this PR implement?
2. How does it work? What's the approach?
3. What components are affected? (backend, frontend, copilot, executor, etc.)
4. What are the key user-facing behaviors to test?

Step 2: Write test scenarios

Based on the PR analysis, write a test plan to $RESULTS_DIR/test-plan.md:

# Test Plan: PR #{N} — {title}

## Scenarios
1. [Scenario name] — [what to verify]
2. ...

## API Tests (if applicable)
1. [Endpoint] — [expected behavior]
   - Before state: [what to check before]
   - After state: [what to verify changed]

## UI Tests (if applicable)
1. [Page/component] — [interaction to test]
   - Screenshot before: [what to capture]
   - Screenshot after: [what to capture]

## Negative Tests (REQUIRED — at least one per feature)
1. [What should NOT happen] — [how to trigger it]
   - Expected error: [what error message/code]
   - State unchanged: [what to verify did NOT change]

Be critical — include edge cases, error paths, and security checks. Every scenario MUST specify what screenshots to take and what state to verify.

Step 3.0: Claim the testing lock (coordinate parallel agents)

Multiple worktrees share the same host — Docker infra (postgres, redis, clamav), app ports (3000/8006/…), and the test user. Two agents running /pr-test concurrently will corrupt each other's state (connection-pool exhaustion, port binds failing silently, cross-test assertions). Use the root-worktree lock file to take turns.

Lock file contract

Path (always the root worktree so all siblings see it): $REPO_ROOT/.ign.testing.lock

Body (one key=value per line):

holder=<pr-XXXXX-purpose>
pid=<pid-or-"self">
started=<iso8601>
heartbeat=<iso8601, updated every ~2 min>
worktree=<full path>
branch=<branch name>
intent=<one-line description + rough duration>

Claim

LOCK=$REPO_ROOT/.ign.testing.lock
NOW=$(date -u +%Y-%m-%dT%H:%MZ)
STALE_AFTER_MIN=5

if [ -f "$LOCK" ]; then
  HB=$(grep '^heartbeat=' "$LOCK" | cut -d= -f2)
  HB_EPOCH=$(date -j -f '%Y-%m-%dT%H:%MZ' "$HB" +%s 2>/dev/null || date -d "$HB" +%s 2>/dev/null || echo 0)
  AGE_MIN=$(( ( $(date -u +%s) - HB_EPOCH ) / 60 ))
  if [ "$AGE_MIN" -gt "$STALE_AFTER_MIN" ]; then
    echo "WARN: stale lock (${AGE_MIN}m old) — reclaiming"
    cat "$LOCK" | sed 's/^/  stale: /'
  else
    echo "Another agent holds the lock:"; cat "$LOCK"
    echo "Wait until released or resume after $((STALE_AFTER_MIN - AGE_MIN))m."
    exit 1
  fi
fi

cat > "$LOCK" <<EOF
holder=pr-${PR_NUMBER}-e2e
pid=self
started=$NOW
heartbeat=$NOW
worktree=$WORKTREE_PATH
branch=$(cd $WORKTREE_PATH && git branch --show-current)
intent=E2E test PR #${PR_NUMBER}, native mode, ~60min
EOF
echo "Lock claimed"

Heartbeat (MUST run in background during the whole test)

Without a heartbeat a crashed agent keeps the lock forever. Run this as a background process right after claim:

(while true; do
   sleep 120
   [ -f "$LOCK" ] || exit 0   # lock released → exit heartbeat
   perl -i -pe "s/^heartbeat=.*/heartbeat=$(date -u +%Y-%m-%dT%H:%MZ)/" "$LOCK"
 done) &
HEARTBEAT_PID=$!
echo "$HEARTBEAT_PID" > /tmp/pr-test-heartbeat.pid

Release (always — even on failure)

kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock" \
    >> $REPO_ROOT/.ign.testing.log

Use a trap so release runs even on exit 1:

trap 'kill "$HEARTBEAT_PID" 2>/dev/null; rm -f "$LOCK"' EXIT INT TERM

Release the lock AS SOON AS the test run is done

The lock guards test execution, not app lifecycle. Once Step 5 (record results) and Step 6 (post PR comment) are complete, release the lock IMMEDIATELY — even if:

• The native poetry run app / pnpm dev processes are still running so the user can keep poking at the app manually.
• You're leaving docker containers up.
• You're tailing logs for a minute or two.

Keeping the lock held past the test run is the single most common way /pr-test stalls other agents. The app staying up is orthogonal to the lock; don't conflate them. Sibling worktrees running their own /pr-test will kill the stray processes and free the ports themselves (Step 3c/3e-native handle that) — they just need the lock file gone.

Concretely, the sequence at the end of every /pr-test run (success or failure) is:

# 1. Write the final report + post PR comment — done above in Step 5/6.
# 2. Release the lock right now, even if the app is still up.
kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock (app may still be running)" \
    >> $REPO_ROOT/.ign.testing.log
# 3. Optionally leave the app running and note it so the user knows:
echo "Native stack still running on :3000 / :8006 for manual poking. Kill with:"
echo "  pkill -9 -f 'poetry run app'; pkill -9 -f 'next-server|next dev'"

If a sibling agent's /pr-test needs to take over, it'll do the kill+rebuild dance from Step 3c/3e-native on its own — your only job is to not hold the lock file past the end of your test.

Shared status log

$REPO_ROOT/.ign.testing.log is an append-only channel any agent can read/write. Use it for "I'm waiting", "I'm done, resources free", or post-run notes:

echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] <message>" \
    >> $REPO_ROOT/.ign.testing.log

Step 3: Environment setup

3a. Copy .env files from the root worktree

The root worktree ($REPO_ROOT) has the canonical .env files with all API keys. Copy them to the target worktree:

# CRITICAL: .env files are NOT checked into git. They must be copied manually.
cp $REPO_ROOT/autogpt_platform/.env $PLATFORM_DIR/.env
cp $REPO_ROOT/autogpt_platform/backend/.env $BACKEND_DIR/.env
cp $REPO_ROOT/autogpt_platform/frontend/.env $FRONTEND_DIR/.env

A copy from the root worktree is no longer sufficient on a recent dev — auth moved from Supabase to Better Auth (see 3h) and two vars are easy to miss because nothing fails loudly without them, it just 401s later:

• $BACKEND_DIR/.env needs JWT_JWKS_URL — the Better Auth JWKS endpoint the backend verifies tokens against. The localhost:3000 value below is for native mode only. In docker mode it's harmless to have it in .env because docker-compose.platform.yml overrides it with the Compose-reachable http://frontend:3000/api/auth/jwks — but if you ever run the backend against this .env value directly (bypassing Compose), a localhost value inside a container resolves to itself, not the frontend, and every call 401s with no other symptom.
• $FRONTEND_DIR/.env needs its own DATABASE_URL — Better Auth runs inside the Next.js app and talks to Postgres directly, it does not go through the backend. Derive it from $BACKEND_DIR/.env's DB_USER / DB_PASS / DB_PORT / DB_NAME rather than copying frontend/.env.default's placeholder verbatim — the placeholder happens to match the stock local defaults, but if backend/.env's DB_PASS was ever customized (rotated secret, non-default port), copying the placeholder silently points Better Auth at the wrong database instead of the one the rest of the stack actually uses.

# [ -n ... ], not grep -q alone — a present-but-empty JWT_JWKS_URL= would
# otherwise be treated as "already set" and skip the fallback, leaving the
# backend without a JWKS endpoint to verify tokens against.
[ -n "$(grep '^JWT_JWKS_URL=' $BACKEND_DIR/.env | cut -d= -f2-)" ] || echo "JWT_JWKS_URL=http://localhost:3000/api/auth/jwks" >> $BACKEND_DIR/.env  # native mode only — see note above

if [ -n "$(grep '^DATABASE_URL=' $FRONTEND_DIR/.env | cut -d= -f2-)" ]; then
  # grep -q alone matches a present-but-empty DATABASE_URL= too, which would
  # otherwise skip derivation and leave Better Auth pointed at nothing.
  echo "Frontend DATABASE_URL: already set (not touching it)"
else
  # cut -f2 (not -f2-) truncates any value containing '=' (base64 secrets do); -f2- keeps the rest.
  DB_USER=$(grep '^DB_USER=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_PASS=$(grep '^DB_PASS=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_PORT=$(grep '^DB_PORT=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_NAME=$(grep '^DB_NAME=' $BACKEND_DIR/.env | cut -d= -f2-)
  : "${DB_USER:?}" "${DB_PASS:?}" "${DB_PORT:?}" "${DB_NAME:?}"  # fail loudly, not with a silently-empty URL
  # Percent-encode user/pass — a raw '@', '#', '?', '%', or ':' in either would
  # otherwise be misparsed as URL structure instead of credential content.
  # Via env vars, not `jq --arg`, which would put DB_PASS in the process arglist.
  DB_USER_ENC=$(DB_USER_VAL="$DB_USER" jq -rn '$ENV.DB_USER_VAL|@uri')
  DB_PASS_ENC=$(DB_PASS_VAL="$DB_PASS" jq -rn '$ENV.DB_PASS_VAL|@uri')
  echo "DATABASE_URL=postgresql://${DB_USER_ENC}:${DB_PASS_ENC}@localhost:${DB_PORT}/${DB_NAME}" >> $FRONTEND_DIR/.env
  # Reconstructed, not regex-redacted — a password containing '@' would otherwise
  # leak its tail past a naive "redact up to the first @" pattern.
  echo "Frontend DATABASE_URL: postgresql://${DB_USER_ENC}:***@localhost:${DB_PORT}/${DB_NAME}"
fi

3b. Configure copilot authentication

The copilot needs an LLM API to function. Two approaches (try subscription first):

Option 1: Subscription mode (preferred — uses your Claude Max/Pro subscription)

The claude_agent_sdk Python package bundles its own Claude CLI binary — no need to install @anthropic-ai/claude-code via npm. The backend auto-provisions credentials from environment variables on startup.

Run the helper script to extract tokens from your host and auto-update backend/.env (works on macOS, Linux, and Windows/WSL):

# Extracts OAuth tokens and writes CLAUDE_CODE_OAUTH_TOKEN + CLAUDE_CODE_REFRESH_TOKEN into .env
bash $BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env

How it works: The script reads the OAuth token from:

• macOS: system keychain ("Claude Code-credentials")
• Linux/WSL: ~/.claude/.credentials.json
• Windows: %APPDATA%/claude/.credentials.json

It sets CLAUDE_CODE_OAUTH_TOKEN, CLAUDE_CODE_REFRESH_TOKEN, and CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true in the .env file. On container startup, the backend auto-provisions ~/.claude/.credentials.json inside the container from these env vars. The SDK's bundled CLI then authenticates using that file. No claude login, no npm install needed.

Note: The OAuth token expires (~24h). If copilot returns auth errors, re-run the script and restart: $BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env && docker compose up -d copilot_executor

Option 2: OpenRouter API key mode (fallback)

If subscription mode doesn't work, switch to API key mode using OpenRouter:

# In $BACKEND_DIR/.env, ensure these are set:
CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false
CHAT_API_KEY=<value of OPEN_ROUTER_API_KEY from the same .env>
CHAT_BASE_URL=https://openrouter.ai/api/v1
CHAT_USE_CLAUDE_AGENT_SDK=true

Use sed to update these values:

ORKEY=$(grep "^OPEN_ROUTER_API_KEY=" $BACKEND_DIR/.env | cut -d= -f2)
[ -n "$ORKEY" ] || { echo "ERROR: OPEN_ROUTER_API_KEY is missing in $BACKEND_DIR/.env"; exit 1; }
perl -i -pe 's/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false/' $BACKEND_DIR/.env
# Add or update CHAT_API_KEY and CHAT_BASE_URL
grep -q "^CHAT_API_KEY=" $BACKEND_DIR/.env && perl -i -pe "s|^CHAT_API_KEY=.*|CHAT_API_KEY=$ORKEY|" $BACKEND_DIR/.env || echo "CHAT_API_KEY=$ORKEY" >> $BACKEND_DIR/.env
grep -q "^CHAT_BASE_URL=" $BACKEND_DIR/.env && perl -i -pe 's|^CHAT_BASE_URL=.*|CHAT_BASE_URL=https://openrouter.ai/api/v1|' $BACKEND_DIR/.env || echo "CHAT_BASE_URL=https://openrouter.ai/api/v1" >> $BACKEND_DIR/.env

3c. Stop conflicting containers

# Stop any running app containers (keep infra: supabase, redis, rabbitmq, clamav)
docker ps --format "{{.Names}}" | grep -E "rest_server|executor|copilot|websocket|database_manager|scheduler|notification|frontend|migrate" | while read name; do
  docker stop "$name" 2>/dev/null
done

Native mode also: when running the app natively (see 3e-native), kill any stray host processes and free the app ports before starting — otherwise poetry run app and pnpm dev will fail to bind.

Kill by port, not by broad process pattern. A pattern-based pkill -f "python.*backend" (or anything matching by worktree cwd) is too coarse on a host running several worktrees — it has taken out the frontend and a mock server sitting on other ports along with the intended backend process. Target the pid actually holding each port instead — this only kills whoever is bound to that specific port, which is narrower than a pattern match, but it is not worktree isolation: if a sibling worktree's own dev server happens to be using one of these ports (e.g. its frontend also on :3000), this kills that too. lsof tells you who holds the port, not who owns it — a docker-proxy pid there means a compose stack owns it.

# Free app ports one at a time — errors per port are ignored (port may simply
# be unused). `xargs -r` is GNU-only (macOS xargs rejects -r); the `[ -n ]`
# guard below is the portable equivalent.
for port in 3000 8006 8001 8002 8005 8008; do
  pids=$(lsof -ti :$port -sTCP:LISTEN 2>/dev/null)
  [ -n "$pids" ] && kill -9 $pids 2>/dev/null || true
done

3e-native. Run the app natively (PREFERRED for iterative dev)

Native mode runs infra (postgres, supabase, redis, rabbitmq, clamav) in docker but runs the backend and frontend directly on the host. This avoids the 3-8 minute docker compose build cycle on every backend change — code edits are picked up on process restart (seconds) instead of a full image rebuild.

When to prefer native mode (default for this skill):

• Iterative dev/debug loops where you're editing backend or frontend code between test runs
• Any PR that touches Python/TS source but not Dockerfiles, compose config, or infra images
• Fast repro of a failing scenario — restart poetry run app in a couple of seconds

When to prefer docker mode (3e fallback):

• Testing changes to Dockerfile, docker-compose.yml, or base images
• Production-parity smoke tests (exact container env, networking, volumes)
• CI-equivalent runs where you need the exact image that'll ship

Note on 3b (copilot auth): no npm install anywhere. poetry install pulls in claude_agent_sdk, which ships its own Claude CLI binary — available on PATH whenever you run commands via poetry run (native) OR whenever the copilot_executor container is built from its Poetry lockfile (docker). The OAuth token extraction still applies (same refresh_claude_token.sh call).

Preamble: before starting native, run the kill-stray + free-ports block from 3c's "Native mode also" subsection.

1. Start infra only (one-time per session):

cd $PLATFORM_DIR && docker compose --profile local up deps --detach --remove-orphans --build

This brings up postgres/supabase/redis/rabbitmq/clamav and skips all app services.

2. Start the backend natively:

cd $BACKEND_DIR && (poetry run app 2>&1 | tee .ign.application.logs) &

poetry run app spawns all app subprocesses — rest_server, executor, copilot_executor, websocket, scheduler, notification_server, database_manager — inside ONE parent process. No separate containers, no separate terminals. The .ign.application.logs prefix is already gitignored.

3. Wait for the backend on :8006 BEFORE starting the frontend. This ordering matters — the frontend's pnpm dev startup invokes generate-api-queries, which fetches /openapi.json from the backend. If the backend isn't listening yet, pnpm dev fails immediately.

for i in $(seq 1 60); do
  if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8006/docs 2>/dev/null)" = "200" ]; then
    echo "Backend ready"
    break
  fi
  sleep 2
done

4. Start the frontend natively:

cd $FRONTEND_DIR && (pnpm dev 2>&1 | tee .ign.frontend.logs) &

5. Wait for the frontend on :3000:

for i in $(seq 1 60); do
  if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:3000 2>/dev/null)" = "200" ]; then
    echo "Frontend ready"
    break
  fi
  sleep 2
done

Once both are up, skip 3e/3f and go straight to 3g/3h (feature flags / test user creation).

3e. Build and start (docker — fallback)

cd $PLATFORM_DIR && docker compose build --no-cache 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker build failed"; exit 1; fi

cd $PLATFORM_DIR && docker compose up -d 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker compose up failed"; exit 1; fi

Note: If the container appears to be running old code (e.g. missing PR changes), use docker compose build --no-cache to force a full rebuild. Docker BuildKit may sometimes reuse cached COPY layers from a previous build on a different branch.

Expected time: 3-8 minutes for build, 5-10 minutes with --no-cache.

3f. Wait for services to be ready

# Poll until backend and frontend respond
for i in $(seq 1 60); do
  BACKEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8006/docs 2>/dev/null)
  FRONTEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null)
  if [ "$BACKEND" = "200" ] && [ "$FRONTEND" = "200" ]; then
    echo "Services ready"
    break
  fi
  sleep 5
done

3h. Create test user and get auth token

The platform moved off Supabase auth to Better Auth, embedded in the Next.js app at /api/auth/*. Signup and sign-in both go through the frontend now, not Kong on :8000 — and /api/auth/token mints a backend-API JWT from a session cookie, it does not accept credentials directly, so sign-in has to happen first to get that cookie.

Better Auth's default minimum password length is 12 characters — shorter values fail signup with PASSWORD_TOO_SHORT and every step below degrades silently into an empty token unless you check for it.

COOKIE_JAR=$(mktemp)
trap 'rm -f "$COOKIE_JAR"' EXIT  # cleans up on early exit too, not just the happy path
# Via env vars, not `jq --arg`, which would put the password in the process arglist.
AUTH_PAYLOAD=$(PR_TEST_USER_EMAIL="$PR_TEST_USER_EMAIL" PR_TEST_USER_PASSWORD="$PR_TEST_USER_PASSWORD" \
  jq -nc '{email:$ENV.PR_TEST_USER_EMAIL,password:$ENV.PR_TEST_USER_PASSWORD,name:"PR Test User"}')

# Signup (idempotent — a real error body means "already exists" only if you
# check; -d passes the payload as an argument, which leaks the password into
# process listings, so pipe it through stdin with --data-binary @- instead.
# --noproxy guards against an inherited proxy env var routing the password
# through a proxy. --max-time bounds it — without one, a server that accepts
# the connection but never responds hangs setup indefinitely instead of
# reaching the empty-$TOKEN failure check below).
SIGNUP_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -X POST 'http://localhost:3000/api/auth/sign-up/email' \
  -H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNUP_RESULT" | grep -qi '"code"' && echo "Signup: $SIGNUP_RESULT"  # log it — "already exists" and "password too short" look identical downstream otherwise

# Sign in — sets the better-auth.session_token cookie in $COOKIE_JAR.
# Capture the body: a failure here (e.g. account exists with a different
# password) otherwise only shows up as an empty $TOKEN with no explanation.
SIGNIN_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -c "$COOKIE_JAR" -X POST 'http://localhost:3000/api/auth/sign-in/email' \
  -H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNIN_RESULT" | grep -qi '"code"' && echo "Sign-in: $SIGNIN_RESULT"

# Mint a backend-API JWT from the session cookie
TOKEN=$(curl -s -b "$COOKIE_JAR" 'http://localhost:3000/api/auth/token' | jq -r '.token // ""')
[ -n "$TOKEN" ] || { echo "ERROR: auth setup failed — TOKEN is empty. Check password length (min 12 chars), AUTH_ALLOW_NEW_ACCOUNTS, AUTH_SIGNUP_ALLOWLIST, and rate limiting on /api/auth/*."; exit 1; }

Use this token for ALL API calls:

curl -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/...

3i. Disable onboarding for test user

The frontend redirects to /onboarding when the ONBOARDING_COMPLETE step is not in completedSteps. Mark it complete via the backend API so every browser test lands on the real feature UI:

ONBOARDING_RESULT=$(curl -s --max-time 30 -X POST \
  "http://localhost:8006/api/onboarding/step?step=ONBOARDING_COMPLETE" \
  -H "Authorization: Bearer $TOKEN")
echo "Onboarding bypass: $ONBOARDING_RESULT"

# Verify it took effect
ONBOARDING_STATUS=$(curl -s --max-time 30 \
  "http://localhost:8006/api/onboarding/completed" \
  -H "Authorization: Bearer $TOKEN" | jq -r '.is_completed')
echo "Onboarding completed: $ONBOARDING_STATUS"
if [ "$ONBOARDING_STATUS" != "true" ]; then
  echo "ERROR: onboarding bypass failed — browser tests will hit /onboarding instead of the target feature. Investigate before proceeding."
  exit 1
fi

Step 4: Run tests

Service ports reference

Service	Port	URL
Frontend	3000	http://localhost:3000
Backend REST	8006	http://localhost:8006
Supabase Auth (via Kong)	8000	http://localhost:8000
Executor (metrics only)	8002	http://localhost:8002 in native mode; not published by Docker Compose
Copilot Executor (metrics only)	8008	http://localhost:8008 in native mode; not published by Docker Compose
WebSocket	8001	http://localhost:8001
Database Manager	8005	http://localhost:8005 in native mode; not published by Docker Compose. Every method needs Authorization: Bearer <internal key> (backend/util/service_auth.py)
Redis	6379	localhost:6379
RabbitMQ	5672	localhost:5672

API testing

Use curl with the auth token for backend API tests. For EVERY API call that changes state, record before/after values:

# Example: List agents
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/graphs | jq . | head -20

# Example: Create an agent
curl -s -X POST http://localhost:8006/api/graphs \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{...}' | jq .

# Example: Run an agent
curl -s -X POST "http://localhost:8006/api/graphs/{graph_id}/execute" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"data": {...}}'

# Example: Get execution results
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8006/api/graphs/{graph_id}/executions/{exec_id}" | jq .

State verification pattern (use for EVERY state-changing API call):

# 1. Record BEFORE state
BEFORE_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "BEFORE: $BEFORE_STATE"

# 2. Perform the action
ACTION_RESULT=$(curl -s -X POST ... | jq .)
echo "ACTION RESULT: $ACTION_RESULT"

# 3. Record AFTER state
AFTER_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "AFTER: $AFTER_STATE"

# 4. Log the comparison
echo "=== STATE CHANGE VERIFICATION ==="
echo "Before: $BEFORE_STATE"
echo "After: $AFTER_STATE"
echo "Expected change: {describe what should have changed}"

Mock-provider pattern (deterministic, $0)

For timeout/latency/error-handling behavior that would otherwise need a real LLM call, point the OpenAI SDK at a local mock instead — it honours OPENAI_BASE_URL, so a small local Responses-API server can stand in for the provider while everything downstream (executor, credentials, billing) still runs for real. This proved out 4/4 test items at $0 in this run. This covers LLM blocks (providers.py) and the Codex block — the copilot's own LLM calls go through backend/util/clients.py, which passes base_url explicitly and does not read OPENAI_BASE_URL, so this trick doesn't reach copilot chat.

# In $BACKEND_DIR/.env, point the OpenAI provider at a local mock server
# that implements the subset of the Responses API you need (e.g. delayed
# responses to test timeout handling, or a 500 to test error surfacing).
# Restart the backend after changing this — it's read at startup.
OPENAI_BASE_URL=http://localhost:{mock_port}/v1

Use a throwaway/dummy provider credential with the mock, never the system credential — only the transport is faked, so the credential lookup still runs for real and whatever key you configure gets sent as a header to your local mock server. A dummy key also means the mock server's logs (which may end up pasted into a PR comment) can't leak a real one. Aside from the credential, this is safe to use for anything that isn't itself testing model output quality.

In docker mode, localhost inside the backend container isn't reachable from your host-side mock server — point OPENAI_BASE_URL at a Compose-reachable hostname instead (or host.docker.internal with a host-gateway entry), and set it before docker compose up or restart the affected services after changing $BACKEND_DIR/.env.

Keep the mock server's own response timeout short — the OpenAI SDK's default client timeout is 600s, so a hung mock stalls every LLM-backed block for that long instead of failing fast.

Browser testing with agent-browser

Primary tool — use this wherever agent-browser is installed:

# Close any existing session
agent-browser close 2>/dev/null || true

# Use --session-name to persist cookies across navigations
# This means login only needs to happen once per test session
agent-browser --session-name pr-test open 'http://localhost:3000/login' --timeout 15000

# Get interactive elements
agent-browser --session-name pr-test snapshot | grep "textbox\|button"

# Login (read creds from env — set PR_TEST_USER_EMAIL / PR_TEST_USER_PASSWORD or ask the user)
agent-browser --session-name pr-test fill {email_ref} "$PR_TEST_USER_EMAIL"
agent-browser --session-name pr-test fill {password_ref} "$PR_TEST_USER_PASSWORD"
agent-browser --session-name pr-test click {login_button_ref}
sleep 5

# Dismiss cookie banner if present
agent-browser --session-name pr-test click 'text=Accept All' 2>/dev/null || true

# Navigate — cookies are preserved so login persists
agent-browser --session-name pr-test open 'http://localhost:3000/copilot' --timeout 10000

# Take screenshot
agent-browser --session-name pr-test screenshot $RESULTS_DIR/01-page.png

# Interact with elements
agent-browser --session-name pr-test fill {ref} "text"
agent-browser --session-name pr-test press "Enter"
agent-browser --session-name pr-test click {ref}
agent-browser --session-name pr-test click 'text=Button Text'

# Read page content
agent-browser --session-name pr-test snapshot | grep "text:"

Key pages:

• /copilot — CoPilot chat (for testing copilot features)
• /build — Agent builder (for testing block/node features)
• /build?flowID={id} — Specific agent in builder
• /library — Agent library (for testing listing/import features)
• /library/agents/{id} — Agent detail with run history
• /marketplace — Marketplace

Fallback — if agent-browser isn't installed on the host, don't let npx download it. Use @playwright/test instead — the package is already in the frontend's node_modules, but the Chromium binary itself is not; Playwright caches browser binaries separately (~/.cache/ms-playwright/ by default) and nothing installs them automatically. Run pnpm exec playwright install chromium once per host if chromium.launch() fails looking for an executable, then use it for finer control over timing (waitForSelector, explicit timeouts) than agent-browser's CLI gives you:

cd $FRONTEND_DIR  # required — @playwright/test is only declared here, not at the repo root
# Check the installer's own exit status, not grep's — piping through grep to
# drop blank lines would otherwise swallow a real install failure and let
# chromium.launch() below fail later with a much less clear error.
pnpm exec playwright install chromium
[ $? -eq 0 ] || { echo "ERROR: playwright install chromium failed"; exit 1; }

node -e "
const { chromium } = require('@playwright/test');
(async () => {
  const browser = await chromium.launch();
  try {
    const page = await browser.newPage();
    await page.goto('http://localhost:3000/login');
    await page.screenshot({ path: '$RESULTS_DIR/01-login.png' });
  } finally {
    await browser.close();  // otherwise a goto timeout leaks a headless Chromium process
  }
})();
"

Checking logs

Native mode: when running via poetry run app + pnpm dev, all app logs stream to the .ign.*.logs files written by the tee pipes in 3e-native. rest_server, executor, copilot_executor, websocket, scheduler, notification_server, and database_manager are all subprocesses of the single poetry run app parent, so their output is interleaved in .ign.application.logs.

# Backend (all app subprocesses interleaved)
tail -f $BACKEND_DIR/.ign.application.logs

# Frontend (Next.js dev server)
tail -f $FRONTEND_DIR/.ign.frontend.logs

# Filter for errors across either log
grep -iE "error|exception|traceback" $BACKEND_DIR/.ign.application.logs | tail -20
grep -iE "error|exception|traceback" $FRONTEND_DIR/.ign.frontend.logs | tail -20

Docker mode:

# Backend REST server
docker logs autogpt_platform-rest_server-1 2>&1 | tail -30

# Executor (runs agent graphs)
docker logs autogpt_platform-executor-1 2>&1 | tail -30

# Copilot executor (runs copilot chat sessions)
docker logs autogpt_platform-copilot_executor-1 2>&1 | tail -30

# Frontend
docker logs autogpt_platform-frontend-1 2>&1 | tail -30

# Filter for errors
docker logs autogpt_platform-executor-1 2>&1 | grep -i "error\|exception\|traceback" | tail -20

Copilot chat testing

The copilot uses SSE streaming. To test via API:

# Create a session
SESSION_ID=$(curl -s -X POST 'http://localhost:8006/api/chat/sessions' \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{}' | jq -r '.id // .session_id // ""')

# Stream a message (SSE - will stream chunks)
curl -N -X POST "http://localhost:8006/api/chat/sessions/$SESSION_ID/stream" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"message": "Hello, what can you help me with?"}' \
  --max-time 60 2>/dev/null | head -50

Or test via browser (preferred for UI verification):

agent-browser --session-name pr-test open 'http://localhost:3000/copilot' --timeout 10000
# ... fill chat input and press Enter, wait 20-30s for response

Step 5: Record results and take screenshots

Take a screenshot at EVERY significant test step — before and after interactions, on success, and on failure. This is NON-NEGOTIABLE.

Required screenshot pattern for each test scenario:

# BEFORE the action
agent-browser --session-name pr-test screenshot $RESULTS_DIR/{NN}-{scenario}-before.png

# Perform the action...

# AFTER the action
agent-browser --session-name pr-test screenshot $RESULTS_DIR/{NN}-{scenario}-after.png

Naming convention:

# Examples:
# $RESULTS_DIR/01-login-page-before.png
# $RESULTS_DIR/02-login-page-after.png
# $RESULTS_DIR/03-credits-page-before.png
# $RESULTS_DIR/04-credits-purchase-after.png
# $RESULTS_DIR/05-negative-insufficient-credits.png
# $RESULTS_DIR/06-error-state.png

Minimum requirements:

• At least TWO screenshots per test scenario (before + after)
• At least ONE screenshot for each negative test case showing the error state
• If a test fails, screenshot the failure state AND any error logs visible in the UI

Step 6: Show results to user with screenshots

CRITICAL: After all tests complete, you MUST show every screenshot to the user using the Read tool, with an explanation of what each screenshot shows. This is the most important part of the test report — the user needs to visually verify the results.

For each screenshot:

1. Use the Read tool to display the PNG file (Claude can read images)
2. Write a 1-2 sentence explanation below it describing:
   • What page/state is being shown
   • What the screenshot proves (which test scenario it validates)
   • Any notable details visible in the UI

Format the output like this:

### Screenshot 1: {descriptive title}
[Read the PNG file here]

**What it shows:** {1-2 sentence explanation of what this screenshot proves}

---

After showing all screenshots, output a detailed summary table:

#	Scenario	Result	API Evidence	Screenshot Evidence
1	{name}	PASS/FAIL	Before: X, After: Y	01-before.png, 02-after.png
2	...	...	...	...

IMPORTANT: As you show each screenshot and record test results, persist them in shell variables for Step 7:

# Build these variables during Step 6 — they are required by Step 7's script
# NOTE: declare -A requires Bash 4.0+. This is standard on modern systems (macOS ships zsh
# but Homebrew bash is 5.x; Linux typically has bash 5.x). If running on Bash <4, use a
# plain variable with a lookup function instead.
declare -A SCREENSHOT_EXPLANATIONS=(
  ["01-login-page.png"]="Shows the login page loaded successfully with SSO options visible."
  ["02-builder-with-block.png"]="The builder canvas displays the newly added block connected to the trigger."
  # ... one entry per screenshot, using the same explanations you showed the user above
)

TEST_RESULTS_TABLE="| 1 | Login flow | PASS | N/A | 01-login-before.png, 02-login-after.png |
| 2 | Credits purchase | PASS | Before: 100, After: 95 | 03-credits-before.png, 04-credits-after.png |
| 3 | Insufficient credits (negative) | PASS | Credits: 0, rejected | 05-insufficient-credits-error.png |"
# ... one row per test scenario with actual results

Step 7: Post test report as PR comment with screenshots

Post the report with gh pr comment --attach. Each attached file is uploaded to GitHub's own asset store, and any image the body already references by that path is rewritten to point at the uploaded asset. Screenshots never touch a repo branch.

This step is MANDATORY. Every test run MUST post a PR comment — with the screenshots attached, or an image-free INCOMPLETE report when attachment is unavailable (see the fallback below). Never nothing.

Attachments are permanent. A user-attachments asset stays reachable by anyone holding its URL regardless of repo visibility, and deleting the comment does not reliably revoke it — there is no undo. Look at every screenshot for credentials, tokens, or customer data before this step runs.

Requires gh >= 2.99.0 — --attach landed there. Distro packages lag (Fedora's rpm tops out at 2.97.0), so check the version you are actually running before building the body:

# Too-old gh is not fatal here: the report still gets posted, just without
# images, through the same fallback path a failed upload takes (below).
GH_VERSION=$(gh version 2>/dev/null | head -1 | awk '{print $3}')
# Numeric compare, not `sort -V`: BSD sort on macOS may lack -V, and an empty
# substitution would silently read as "too old" on a perfectly good gh.
GH_MAJOR=${GH_VERSION%%.*}; GH_REST=${GH_VERSION#*.}; GH_MINOR=${GH_REST%%.*}
ATTACH_OK=0
ATTACH_SKIP_REASON=""
if [ "${GH_MAJOR:-0}" -gt 2 ] 2>/dev/null || { [ "${GH_MAJOR:-0}" -eq 2 ] && [ "${GH_MINOR:-0}" -ge 99 ]; } 2>/dev/null; then
  ATTACH_OK=1
else
  ATTACH_SKIP_REASON="gh ${GH_VERSION:-not found} is below 2.99.0 — upgrade gh"
  echo "WARN: ${ATTACH_SKIP_REASON}; posting the report without images."
fi

CRITICAL — NEVER post a bare directory link like https://github.com/.../tree/.... Every screenshot MUST appear as ![name](url) inline in the PR comment so reviewers can see them without clicking any links. After posting, the verification step below counts the rewritten asset URLs against the number of screenshots and exits 1 on any shortfall — the test run is incomplete until this passes.

CRITICAL — NEVER paste absolute local paths or credentials into the PR comment. Strings like /Users/…, /home/…, C:\… are useless to every reviewer except you, and a bearer token or JWT copied from the live stack's evidence is a real leak in a public comment. The build block below runs an egress check on the exact bytes about to be posted and aborts on either. The ./01-shot.png references the attachments use are relative, not local paths, and are rewritten on upload — they are fine. Keep local paths in $RESULTS_DIR/test-report.md for yourself; only copy the content they reference (excerpts, test names, log lines) into the PR comment, not the path.

Build the body with relative image references, then attach the same paths:

REPO="Significant-Gravitas/AutoGPT"
MAX_ATTACHMENTS=50   # per gh pr comment invocation

# Fail closed: an empty RESULTS_DIR would glob the PR worktree's PNGs into a public comment.
# gh matches --attach to the body by exact path, so the cd is required; undone at the end.
[ -n "$RESULTS_DIR" ] && [ -d "$RESULTS_DIR" ] || { echo "ERROR: RESULTS_DIR is unset or not a directory: '$RESULTS_DIR'"; exit 1; }
STEP7_ORIG_DIR=$PWD
cd "$RESULTS_DIR" || exit 1
NULLGLOB_WAS=$(shopt -p nullglob)   # restore the caller's setting, don't force it off
shopt -s nullglob
SCREENSHOT_FILES=(*.png)
$NULLGLOB_WAS
if [ ${#SCREENSHOT_FILES[@]} -eq 0 ]; then
  echo "ERROR: No screenshots found in $RESULTS_DIR. Test run is incomplete."
  exit 1
fi
# Over the cap, post the report without images rather than nothing at all.
if [ ${#SCREENSHOT_FILES[@]} -gt "$MAX_ATTACHMENTS" ]; then
  ATTACH_SKIP_REASON="${#SCREENSHOT_FILES[@]} screenshots exceed the ${MAX_ATTACHMENTS}-attachment limit per comment"
  echo "WARN: ${ATTACH_SKIP_REASON}; posting the report without images."
  ATTACH_OK=0
fi

IMAGE_MARKDOWN=""
ATTACH_ARGS=()
for BASENAME in "${SCREENSHOT_FILES[@]}"; do
  TITLE=$(echo "${BASENAME%.png}" | sed 's/^[0-9]*-//' | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')
  EXPLANATION="${SCREENSHOT_EXPLANATIONS[$BASENAME]}"
  if [ -z "$EXPLANATION" ]; then
    echo "ERROR: Missing screenshot explanation for $BASENAME. Add it to SCREENSHOT_EXPLANATIONS in Step 6."
    exit 1
  fi
  IMAGE_MARKDOWN="${IMAGE_MARKDOWN}
### ${TITLE}
![${TITLE}](./${BASENAME})
${EXPLANATION}
"
  # Alt text after '#' is the fallback; the body reference above wins when both exist.
  ATTACH_ARGS+=(--attach "./${BASENAME}#${TITLE}")
done

# Keep the report and the image section separate: the no-image fallback below
# must not carry "![](./file.png)" references it has no attachments for.
RUN_MARKER="<!-- pr-test-report:${PR_NUMBER}:$(date -u +%Y%m%dT%H%M%SZ) -->"
REPORT_BODY="${RUN_MARKER}
## E2E Test Report

| # | Scenario | Result | API Evidence | Screenshot Evidence |
|---|----------|--------|-------------|-------------------|
${TEST_RESULTS_TABLE}
"

COMMENT_FILE=$(mktemp)
printf '%s\n%s\n' "$REPORT_BODY" "$IMAGE_MARKDOWN" > "$COMMENT_FILE"

# Egress check on the exact bytes about to be posted. Fails closed: a missing
# body is an abort, not a pass.
LEAK_RE='(^|[^A-Za-z])(/Users/|/home/|/tmp/|/private/|C:\\|~/)[A-Za-z0-9]|eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{10,}|(sk|ghp|gho|github_pat)_[A-Za-z0-9_]{16,}|[Bb]earer [A-Za-z0-9._-]{20,}'
[ -s "$COMMENT_FILE" ] || { echo "ABORT: comment body missing or empty."; exit 1; }
if grep -nE "$LEAK_RE" "$COMMENT_FILE"; then
  echo "ABORT: local paths or credential-shaped strings in the PR comment body. Rewrite them before posting."
  exit 1
fi

# A blind retry duplicates a post GitHub accepted but gh reported failed; the marker
# makes "did it land?" checkable first. Real jq (gh's --jq has no --arg); --paginate
# because the per-issue endpoint ignores sort/direction.
URL_RE='https://[^[:space:]]+#issuecomment-[0-9]+'
POSTED=""
if [ "$ATTACH_OK" = 1 ]; then
  for attempt in 1 2 3; do
    # The marker was minted seconds ago, so attempt 1 cannot already be posted.
    # A lookup that errors is "unknown", not "not found": posting on unknown is
    # how the duplicate the marker exists to prevent would get made.
    if [ "$attempt" -gt 1 ]; then
      if ! FOUND=$(set -o pipefail; gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" --paginate \
          | jq -r --arg m "$RUN_MARKER" 'first(.[] | select(.body | contains($m)) | .html_url) // empty'); then
        echo "Attempt $attempt: marker lookup failed; not posting until it can be confirmed"
        sleep $((attempt * attempt * 2)); continue
      fi
      POSTED=${FOUND%%$'\n'*}
      [ -n "$POSTED" ] && break
    fi
    # Take the URL by shape, not position: gh prints its update notice on stderr at
    # exit, and a trailing one would otherwise ride into COMMENT_ID below.
    OUT=$(gh pr comment "$PR_NUMBER" --repo "$REPO" --body-file "$COMMENT_FILE" "${ATTACH_ARGS[@]}" 2>&1)
    POSTED=$(printf '%s' "$OUT" | grep -oE "$URL_RE" | tail -1)
    [ -n "$POSTED" ] && break
    echo "Attempt $attempt failed: $OUT"
    [ "$attempt" -lt 3 ] && sleep $((attempt * attempt * 2))   # 2s, 8s — secondary rate limits need room
  done
fi

If gh is too old, the run is over the cap, or all three attempts failed, post the report without images and say so, so the run is visibly incomplete rather than silently missing its evidence. The run continues into Step 8 — an image-free report is not approvable, and that is Step 8's job to say:

LOOKUP_UNKNOWN=0
if [ -z "$POSTED" ] && [ "$ATTACH_OK" = 1 ]; then
  # One last marker check — a final attempt may have landed despite reporting failure.
  if FOUND=$(set -o pipefail; gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" --paginate \
      | jq -r --arg m "$RUN_MARKER" 'first(.[] | select(.body | contains($m)) | .html_url) // empty'); then
    POSTED=${FOUND%%$'\n'*}
  else
    LOOKUP_UNKNOWN=1
  fi
fi
RUN_INCOMPLETE=0
if [ -z "$POSTED" ] && [ "$LOOKUP_UNKNOWN" = 1 ]; then
  # Can't tell whether the report landed; a fallback post here risks a duplicate.
  RUN_INCOMPLETE=1
  echo "ERROR: could not confirm whether the report was posted (GitHub API errors). Check the PR for marker ${RUN_MARKER} before re-running Step 7."
elif [ -z "$POSTED" ]; then
  RUN_INCOMPLETE=1
  [ "$ATTACH_OK" = 1 ] && ATTACH_SKIP_REASON="the upload failed 3 times"
  FALLBACK_FILE=$(mktemp)
  {
    printf '%s\n' "$REPORT_BODY"
    printf '## ⚠️ Screenshots not attached\n\n'
    printf '%s. Filenames, for manual drag-and-drop into a reply:\n\n' "$ATTACH_SKIP_REASON"
    printf -- '- `%s`\n' "${SCREENSHOT_FILES[@]}"
    printf '\n**Run status:** INCOMPLETE until the files are attached and visible inline in the PR.\n'
  } > "$FALLBACK_FILE"
  # The degraded path gets the same egress check as the full body.
  if grep -nE "$LEAK_RE" "$FALLBACK_FILE"; then
    echo "ABORT: local paths or credential-shaped strings in the fallback body."
    exit 1
  fi
  OUT=$(gh pr comment "$PR_NUMBER" --repo "$REPO" --body-file "$FALLBACK_FILE" 2>&1)
  POSTED=$(printf '%s' "$OUT" | grep -oE "$URL_RE" | tail -1)
  rm -f "$FALLBACK_FILE"
  echo "Posted without images (RUN_INCOMPLETE=1): ${POSTED:-FAILED — $OUT}"
fi
rm -f "$COMMENT_FILE"
cd "$STEP7_ORIG_DIR"

Verify the rendered comment actually carries every screenshot as an uploaded asset (skipped for an image-free fallback, which Step 8 already treats as not approvable):

if [ "$RUN_INCOMPLETE" = 0 ]; then
  COMMENT_ID="${POSTED##*issuecomment-}"
  BODY=$(gh api "repos/${REPO}/issues/comments/${COMMENT_ID}" --jq '.body')
  EXPECTED=${#SCREENSHOT_FILES[@]}
  # Count, don't just detect: a partial rewrite leaves broken "](./x.png)"
  # links behind while the body still contains *some* asset URLs.
  UPLOADED=$(printf '%s' "$BODY" | grep -o 'github.com/user-attachments/' | wc -l | tr -d ' ')
  LEFTOVER=$(printf '%s' "$BODY" | grep -o '](\./' | wc -l | tr -d ' ')
  RAW_IMGS=$(printf '%s' "$BODY" | grep -oE '!\[[^]]*\]\(https://raw\.githubusercontent\.com' | wc -l | tr -d ' ')
  if [ "$RAW_IMGS" -gt 0 ]; then
    echo "ERROR: $RAW_IMGS image(s) still point at raw repo URLs. Screenshots must be attachments, not files in the repo." >&2
    exit 1
  fi
  if [ "$LEFTOVER" -gt 0 ] || [ "$UPLOADED" -ne "$EXPECTED" ]; then
    echo "ERROR: expected $EXPECTED uploaded attachments, found $UPLOADED, with $LEFTOVER unrewritten ./ reference(s) — partial upload." >&2
    exit 1
  fi
  echo "✓ $EXPECTED screenshots verified as GitHub attachments"
fi

The PR comment MUST include:

1. A summary table of all scenarios with PASS/FAIL and before/after API evidence
2. Every screenshot rendered inline as an uploaded attachment; any failure listed with manual attachment instructions and the run marked INCOMPLETE
3. A 1-2 sentence explanation below each screenshot describing what it proves

Step 8: Evaluate and post a formal PR review

After the test comment is posted, evaluate whether the run was thorough enough to make a merge decision, then post a formal GitHub review (approve or request changes). This step is mandatory — every test run MUST end with a formal review decision.

Evaluation criteria

Re-read the PR description:

gh pr view "$PR_NUMBER" --json body --jq '.body' --repo "$REPO"

Score the run against each criterion:

Criterion	Pass condition
Coverage	Every feature/change described in the PR has at least one test scenario
All scenarios pass	No FAIL rows in the results table
Negative tests	At least one failure-path test per feature (invalid input, unauthorized, edge case)
Before/after evidence	Every state-changing API call has before/after values logged
Screenshots are meaningful	Screenshots show the actual state change, not just a loading spinner or blank page
No regressions	Existing core flows (login, agent create/run) still work

Decision logic

ALL criteria pass                            → APPROVE
Any scenario FAIL or missing PR feature      → REQUEST_CHANGES (list gaps)
Evidence weak (no before/after, vague shots) → REQUEST_CHANGES (list what's missing)
Screenshots not attached (RUN_INCOMPLETE=1)  → REQUEST_CHANGES (the evidence is not on the PR)

Post the review

REVIEW_FILE=$(mktemp)

# Count results
PASS_COUNT=$(echo "$TEST_RESULTS_TABLE" | grep -c "PASS" || true)
FAIL_COUNT=$(echo "$TEST_RESULTS_TABLE" | grep -c "FAIL" || true)
TOTAL=$(( PASS_COUNT + FAIL_COUNT ))

# List any coverage gaps found during evaluation (populate this array as you assess)
# e.g. COVERAGE_GAPS=("PR claims to add X but no test covers it")
COVERAGE_GAPS=()
# Step 7 posts an image-free report instead of aborting; that run is not approvable.
[ "${RUN_INCOMPLETE:-0}" = 1 ] && COVERAGE_GAPS+=("Screenshots were not attached to the test report — attach them and re-run verification")

If APPROVING — all criteria met, zero failures, full coverage:

cat > "$REVIEW_FILE" <<REVIEWEOF
## E2E Test Evaluation — APPROVED

**Results:** ${PASS_COUNT}/${TOTAL} scenarios passed.

**Coverage:** All features described in the PR were exercised.

**Evidence:** Before/after API values logged for all state-changing operations; screenshots show meaningful state transitions.

**Negative tests:** Failure paths tested for each feature.

No regressions observed on core flows.
REVIEWEOF

gh pr review "$PR_NUMBER" --repo "$REPO" --approve --body "$(cat "$REVIEW_FILE")"
echo "✅ PR approved"

If REQUESTING CHANGES — any failure, coverage gap, or missing evidence:

FAIL_LIST=$(echo "$TEST_RESULTS_TABLE" | grep "FAIL" | awk -F'|' '{print "- Scenario" $2 "failed"}' || true)

cat > "$REVIEW_FILE" <<REVIEWEOF
## E2E Test Evaluation — Changes Requested

**Results:** ${PASS_COUNT}/${TOTAL} scenarios passed, ${FAIL_COUNT} failed.

### Required before merge

${FAIL_LIST}
$(for gap in "${COVERAGE_GAPS[@]}"; do echo "- $gap"; done)

Please fix the above and re-run the E2E tests.
REVIEWEOF

gh pr review "$PR_NUMBER" --repo "$REPO" --request-changes --body "$(cat "$REVIEW_FILE")"
echo "❌ Changes requested"

rm -f "$REVIEW_FILE"

Rules:

• In --fix mode, fix all failures before posting the review — the review reflects the final state after fixes
• Never approve if any scenario failed, even if it seems like a flake — rerun that scenario first
• Never approve a run with RUN_INCOMPLETE=1 — the evidence is not on the PR
• Never request changes for issues already fixed in this run

Fix mode (--fix flag)

When --fix is present, the standard is HIGHER. Do not just note issues — FIX them immediately.

Fix protocol for EVERY issue found (including UX issues):

1. Identify the root cause in the code — read the relevant source files
2. Write a failing test first (TDD): For backend bugs, write a test marked with pytest.mark.xfail(reason="..."). For frontend/Playwright bugs, write a test with .fixme annotation. Run it to confirm it fails as expected.
3. Screenshot the broken state: agent-browser screenshot $RESULTS_DIR/{NN}-broken-{description}.png
4. Fix the code in the worktree
5. Rebuild ONLY the affected service (not the whole stack):

   cd $PLATFORM_DIR && docker compose up --build -d {service_name}
   # e.g., docker compose up --build -d rest_server
   # e.g., docker compose up --build -d frontend
   

6. Wait for the service to be ready (poll health endpoint)
7. Re-test the same scenario
8. Screenshot the fixed state: agent-browser screenshot $RESULTS_DIR/{NN}-fixed-{description}.png
9. Remove the xfail/fixme marker from the test written in step 2, and verify it passes
10. Verify the fix did not break other scenarios (run a quick smoke test)
11. Commit and push immediately:

cd $WORKTREE_PATH
git add -A
git commit -m "fix: {description of fix}"
git push

12. Continue to the next test scenario

Fix loop (like pr-address)

test scenario → find issue (bug OR UX problem) → screenshot broken state
→ fix code → rebuild affected service only → re-test → screenshot fixed state
→ verify no regressions → commit + push
→ repeat for next scenario
→ after ALL scenarios pass, run full re-test to verify everything together

Key differences from non-fix mode:

• UX issues count as bugs — fix them (bad alignment, confusing labels, missing loading states)
• Every fix MUST have a before/after screenshot pair proving it works
• Commit after EACH fix, not in a batch at the end
• The final re-test must produce a clean set of all-passing screenshots

Known issues and workarounds

Problem: Signup/sign-in fails with no useful error

Cause: Better Auth's default minimum password length is 12 characters, or AUTH_ALLOW_NEW_ACCOUNTS=false / AUTH_SIGNUP_ALLOWLIST is blocking new accounts, or you're rate-limited after repeated signup attempts (429 Too many requests) — all three degrade into an empty $TOKEN further down if you don't check the raw response (see 3h). Fix: Log the raw signup/sign-in response body before minting the token, not just the token itself.

Problem: Copilot returns auth errors in subscription mode

Cause: CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true but CLAUDE_CODE_OAUTH_TOKEN is not set or expired. Fix: Re-extract the OAuth token from macOS keychain (see step 3b, Option 1) and recreate the container (docker compose up -d copilot_executor). The backend auto-provisions ~/.claude/.credentials.json from the env var on startup. No npm install or claude login needed — the SDK bundles its own CLI binary.

Problem: agent-browser can't find chromium

Cause: The Dockerfile auto-provisions system chromium on all architectures (including ARM64). If your branch is behind dev, this may not be present yet. Fix: Check if chromium exists: which chromium || which chromium-browser. If missing, install it: apt-get install -y chromium and set AGENT_BROWSER_EXECUTABLE_PATH=/usr/bin/chromium in the container environment.

Problem: agent-browser selector matches multiple elements

Cause: text=X matches all elements containing that text. Fix: Use agent-browser snapshot to get specific ref=eNN references, then use those: agent-browser click eNN.

Problem: Frontend shows cookie banner blocking interaction

Fix: agent-browser click 'text=Accept All' before other interactions.

Problem: Claude CLI not found in copilot_executor container

Symptom: Copilot logs say claude: command not found or similar when starting an SDK turn. Cause: Image was built without poetry install (stale base layer, or Dockerfile bypass). The SDK CLI ships inside the claude_agent_sdk Poetry dep — it is NOT an npm package. Fix: Rebuild the image cleanly: docker compose build --no-cache copilot_executor && docker compose up -d copilot_executor. Do NOT docker exec ... npm install -g @anthropic-ai/claude-code — that is outdated guidance and will pollute the container with a second CLI that the SDK won't use.

Problem: agent-browser screenshot hangs / times out

Symptom: agent-browser screenshot exits with code 124 even on about:blank. Cause: Stuck CDP connection or Chromium process tree. Seen on macOS when a prior /pr-test left a zombie Chrome for Testing. Fix: pkill -9 -f "agent-browser|chromium|Chrome for Testing" && sleep 2, then reopen the browser with a fresh --session-name. If still failing, verify via agent-browser eval + agent-browser snapshot (DOM state) instead of relying on PNGs — the feature under test is the same.

Problem: Services not starting after docker compose up

Fix: Wait and check health: docker compose ps. Common cause: migration hasn't finished. Check: docker logs autogpt_platform-migrate-1 2>&1 | tail -5. If the db container isn't healthy: docker restart autogpt_platform-db-1 && sleep 10.

Problem: Docker uses cached layers with old code (PR changes not visible)

Cause: docker compose up --build reuses cached COPY layers from previous builds. If the PR branch changes Python files but the previous build already cached that layer from dev, the container runs dev code. Fix: Always use docker compose build --no-cache for the first build of a PR branch. Subsequent rebuilds within the same branch can use --build.

Problem: agent-browser open loses login session

Cause: Without session persistence, agent-browser open starts fresh. Fix: Use --session-name pr-test on ALL agent-browser commands. This auto-saves/restores cookies and localStorage across navigations. Alternatively, use agent-browser eval "window.location.href = '...'" to navigate within the same context.