Test — Run All Tests: Unit → Integration → E2E

Auto-detect all test layers (unit, integration, frontend, backend, E2E/system), run them all, write more if coverage is below threshold, loop until everything passes.

Do NOT ask for permission. Do NOT stop until coverage is met.

---

Testing philosophy

Prefer real implementations over mocks.

Layer	Environment	What to mock
Unit	In-process only	External APIs that cannot run locally (Stripe, Firebase Auth, Resend, Twilio, cloud SDKs)
Integration	Real DB + real app via Docker	Only payment gateways and third-party external APIs
E2E / System	Full stack via docker compose up	Nothing — zero mocks

Never mock: the database, internal services, business logic, utilities, or pure functions.
Only mock: services that are genuinely unreachable locally (payment processors, auth providers, email senders, external SaaS APIs).

---

How to use

• /test — all layers for files changed since last commit
• /test --all — full pass across the entire codebase
• /test <file> — target a specific source file
• /test --unit — unit tests only
• /test --integration — integration tests only (spins up Docker services)
• /test --e2e — E2E/system tests only (full docker compose stack)
• /test --e2e staging — E2E against staging environment
• /test --e2e prod — E2E against production

---

Coverage thresholds (unit + integration — not E2E)

Defaults below apply only when the project does not declare its own — check first:

# Project-declared thresholds win over the defaults below. Look, in order, at:
#   - the instruction file (CLAUDE.md/AGENTS.md/…): a line like "coverage: lines 80, branches 70"
#   - jest/vitest config: coverageThreshold.global.{lines,functions,statements,branches}
#   - pyproject.toml / .coveragerc: [tool.coverage.report] fail_under
grep -hiE "coverage(Threshold)?|fail_under" CLAUDE.md AGENTS.md jest.config.* vitest.config.* vite.config.* pyproject.toml .coveragerc 2>/dev/null | head

Metric	Default threshold (project-declared value overrides)
Lines	≥ 95%
Functions	≥ 95%
Statements	≥ 95%
Branches	≥ 90%

The coverage loop does not exit until ALL thresholds are met AND zero unit/integration tests fail — or the iteration cap is reached (see Phase 3).

---

Phase 0 — Docker test environment setup

Before integration or E2E tests, check if Docker services are required and start them.

PROJECT_ROOT=$(git rev-parse --show-toplevel)
cd "$PROJECT_ROOT"

Detect test service requirements

# Check for docker-compose test config
TEST_COMPOSE=""
for f in docker-compose.test.yml docker-compose.testing.yml docker-compose.yml compose.yml; do
  [ -f "$PROJECT_ROOT/$f" ] && TEST_COMPOSE="$f" && break
done

# Check for service dependencies in pyproject.toml or package.json
grep -qE "postgres|redis|mysql|mongodb|elasticsearch" \
  "$PROJECT_ROOT/pyproject.toml" \
  "$PROJECT_ROOT/requirements*.txt" \
  "$PROJECT_ROOT/package.json" \
  "$PROJECT_ROOT/api/package.json" 2>/dev/null && NEEDS_SERVICES=true || NEEDS_SERVICES=false

echo "Test compose file: ${TEST_COMPOSE:-none}"
echo "Needs services: $NEEDS_SERVICES"

Start test services (if needed)

# Auto-detect and start required services
TEST_COMPOSE=$(ls docker-compose.test.yml docker-compose.testing.yml docker-compose.yml compose.yml 2>/dev/null | head -1 || true)
NEEDS_SERVICES=$(grep -qE "postgres|redis|mysql|mongodb|elasticsearch" \
  "$PROJECT_ROOT/pyproject.toml" "$PROJECT_ROOT/requirements"*.txt "$PROJECT_ROOT/package.json" 2>/dev/null && echo true || echo false)

if [ -n "$TEST_COMPOSE" ]; then
  docker compose -f "$TEST_COMPOSE" up -d --wait 2>/dev/null || true
elif $NEEDS_SERVICES; then
  # Match the version the project already declares — don't pin a version it doesn't use.
  # Look in any compose file for the image tag; fall back to a current LTS only if absent.
  PG_IMAGE=$(grep -hoE "postgres:[0-9.]+(-[a-z0-9]+)?" "$PROJECT_ROOT"/docker-compose*.y*ml "$PROJECT_ROOT"/compose*.y*ml 2>/dev/null | head -1)
  REDIS_IMAGE=$(grep -hoE "redis:[0-9.]+(-[a-z0-9]+)?" "$PROJECT_ROOT"/docker-compose*.y*ml "$PROJECT_ROOT"/compose*.y*ml 2>/dev/null | head -1)
  : "${PG_IMAGE:=postgres:16}"; : "${REDIS_IMAGE:=redis:7}"   # defaults only when undeclared
  docker run -d --name test-postgres \
    -e POSTGRES_USER=test -e POSTGRES_PASSWORD=test -e POSTGRES_DB=test \
    -p 5432:5432 "$PG_IMAGE" 2>/dev/null || true
  grep -qE "redis" "$PROJECT_ROOT/package.json" "$PROJECT_ROOT/pyproject.toml" 2>/dev/null && \
    docker run -d --name test-redis -p 6379:6379 "$REDIS_IMAGE" 2>/dev/null || true
  sleep 3 && docker exec test-postgres pg_isready -U test 2>/dev/null || sleep 3
fi

Run migrations against test DB:

# Django
[ -f manage.py ] && ./venv/bin/python manage.py migrate --settings=config.settings.test 2>/dev/null || true
# Alembic
[ -f alembic.ini ] && ./venv/bin/alembic upgrade head 2>/dev/null || true
# Prisma
[ -f prisma/schema.prisma ] && npx prisma migrate deploy 2>/dev/null || true
# Custom
[ -f scripts/migrate.sh ] && bash scripts/migrate.sh 2>/dev/null || true

Verify services are healthy before proceeding:

docker compose ps 2>/dev/null || docker ps --filter "name=test-" 2>/dev/null

GATE: Required services must be running before integration or E2E tests.

Cleanup (run after all tests complete)

# Stop test compose
[ -n "$TEST_COMPOSE" ] && docker compose -f "$TEST_COMPOSE" down -v 2>/dev/null || true
# Stop standalone containers
docker rm -f test-postgres test-redis 2>/dev/null || true

---

Step 1 — Detect test stack

ls package.json api/package.json requirements.txt pyproject.toml 2>/dev/null
cat package.json 2>/dev/null | grep -E '"vitest"|"jest"' || true
cat api/package.json 2>/dev/null | grep '"jest"' || true
ls e2e/ tests/e2e/ playwright.config.* e2e/playwright.config.* 2>/dev/null || true

Determine which layers apply:

• Frontend unit/integration (Vitest): root package.json has vitest
• Backend unit/integration (Jest): api/package.json has jest
• Python unit/integration (pytest): pyproject.toml or requirements.txt present
• E2E/System (Playwright): playwright.config.* found anywhere
• E2E/System (pytest): tests/e2e/ directory found in Python project

---

Phase 1 — Unit Tests

Run smallest-scope tests first for fast feedback.

The detected unit layers (Frontend Vitest / Backend Jest / Python pytest) share no state — fan them out per the subagents skill ladder (Workflow tool → parallel subagents → serial fallback). Each layer returns { layer, status, passed, failed, coverage }; the parent collects them. Safe before Phase 2 only — the integration phase shares one Docker DB and stays serial.

Frontend unit (Vitest):

cd "$PROJECT_ROOT"
npm run test:unit 2>&1

Backend unit (Jest):

cd "$PROJECT_ROOT/api"
npm run test:unit 2>&1

Python unit (pytest):

cd "$PROJECT_ROOT"
./venv/bin/python -m pytest tests/unit/ -v --tb=short -q

Test patterns:

• Pure functions, hooks, utilities, lib modules
• Every code path: success, error, edge cases, empty input, boundary values
• Mock ONLY genuinely unreachable external services: Stripe, Firebase Auth, Resend, Twilio, AWS SES
• Do NOT mock the database — unit tests needing DB state use the real test DB from Phase 0

---

Phase 2 — Integration Tests

Run against real services started in Phase 0. No mocking of internal infrastructure.

Frontend integration (Vitest):

cd "$PROJECT_ROOT"
npm run test:integration 2>&1

Backend integration (Jest + supertest):

cd "$PROJECT_ROOT/api"
# Set test DB URL — real Docker DB from Phase 0
DATABASE_URL="${TEST_DATABASE_URL:-postgresql://test:test@localhost:5432/test}" \
REDIS_URL="${TEST_REDIS_URL:-redis://localhost:6379}" \
npm run test:integration 2>&1

Python integration (pytest):

cd "$PROJECT_ROOT"
DATABASE_URL="${TEST_DATABASE_URL:-postgresql+asyncpg://test:test@localhost:5432/test}" \
REDIS_URL="${TEST_REDIS_URL:-redis://localhost:6379}" \
./venv/bin/python -m pytest tests/integration/ -v --tb=short -q

Test patterns:

• Full HTTP request → response through the real app against a real DB
• Multi-component flows, context providers, routing, auth state
• Real DB reads and writes — assert actual persisted state, not mock return values
• Do NOT mock: your own DB, Redis, internal queues, internal services

---

Phase 3 — Coverage loop (unit + integration)

Run coverage for all detected stacks; loop until thresholds are met:

Vitest (frontend):

cd "$PROJECT_ROOT"
npm run test:coverage 2>&1

Jest (backend):

cd "$PROJECT_ROOT/api"
DATABASE_URL="${TEST_DATABASE_URL:-postgresql://test:test@localhost:5432/test}" \
npm run test:coverage 2>&1

pytest (Python):

cd "$PROJECT_ROOT"
DATABASE_URL="${TEST_DATABASE_URL:-postgresql+asyncpg://test:test@localhost:5432/test}" \
./venv/bin/python -m pytest tests/unit/ tests/integration/ --cov=. --cov-report=term-missing -q

Loop logic (bounded — never spin forever):

1. Parse coverage output — find files below threshold, find failing tests
2. If all thresholds met AND zero failures → exit loop ✅
3. Otherwise, if fewer than MAX_ITERATIONS (default 6) iterations have run:
   • For each uncovered file: read it, write tests targeting uncovered lines/branches
   • For each failing test: fix the test or the underlying code
   • Re-run from top of loop
4. If the cap is reached without converging → STOP and escalate. Report the remaining gap (files/metrics short, tests still failing) and ask the user whether to keep going, lower a threshold, or investigate a stuck test. Never loop indefinitely on an unreachable target (e.g. coverage blocked by an untestable external dependency).

Rules inside the loop:

• Read the source file before writing tests — understand all code paths
• Test ALL paths: success, error, edge cases, auth failures, DB errors, empty state
• Never skip, xfail, or comment-out failing tests — fix the code or the test
• Each iteration targets the lowest-coverage files first

---

Phase 4 — E2E / System Tests (Docker full-stack)

Run after unit + integration pass. Spin up the complete stack via Docker and run real browser or API tests against it. Zero mocks.

4a. Split Playwright configs

Use two separate Playwright config files — component/browser-unit tests (no full stack) vs true E2E against docker compose:

playwright.config.ts          ← component tests (no server required)
playwright.e2e.config.ts      ← full-stack E2E (requires docker compose)
e2e/
  fixtures/
    auth.ts                   ← real auth fixture (creates user, gets JWT)
    api.ts                    ← API helper (authenticated HTTP client)
  smoke/
    health.spec.ts            ← health + smoke tests
  auth/
    login.spec.ts             ← real browser login flow
  agents/
    crud.spec.ts              ← real CRUD tests via browser

playwright.config.ts (component tests — no docker required):

import { defineConfig } from '@playwright/test'

export default defineConfig({
  testDir: './src',
  testMatch: '**/*.spec.ts',
  use: { baseURL: 'http://localhost:5173' },
  webServer: {
    command: 'npm run dev',
    port: 5173,
    reuseExistingServer: !process.env.CI,
  },
})

playwright.e2e.config.ts (full-stack E2E — requires docker compose up):

import { defineConfig } from '@playwright/test'

export default defineConfig({
  testDir: './e2e',
  testMatch: '**/*.spec.ts',
  timeout: 60_000,
  retries: process.env.CI ? 2 : 0,
  reporter: [['html', { open: 'never' }], ['list']],
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
})

4b. Docker Compose E2E override file

Create docker-compose.e2e.yml overriding production compose settings for E2E (seed data, test credentials, exposed ports):

# docker-compose.e2e.yml
# Extend your base docker-compose.yml for E2E tests
# Usage: docker compose -f docker-compose.yml -f docker-compose.e2e.yml up -d

services:
  api:
    environment:
      - NODE_ENV=test
      - DATABASE_URL=postgresql://test:test@db:5432/testdb
      - REDIS_URL=redis://redis:6379
      - JWT_SECRET=e2e-test-secret-not-for-production
      - SEED_TEST_DATA=true          # trigger seed on startup
    ports:
      - "8000:8000"

  web:
    environment:
      - VITE_API_URL=http://localhost:8000
    ports:
      - "3000:3000"

  db:
    image: postgres:16          # match your base docker-compose.yml version — don't drift the test DB
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: testdb
    ports:
      - "5432:5432"
    tmpfs:
      - /var/lib/postgresql/data   # ephemeral — fast, discarded after tests

  redis:
    image: redis:7               # match your base docker-compose.yml version
    ports:
      - "6379:6379"

Detection and start:

cd "$PROJECT_ROOT"

# Prefer e2e override, fall back to test compose, fall back to main
if [ -f docker-compose.e2e.yml ] && [ -f docker-compose.yml ]; then
  COMPOSE_CMD="docker compose -f docker-compose.yml -f docker-compose.e2e.yml"
  echo "Starting E2E stack with override: docker-compose.e2e.yml"
elif [ -f docker-compose.test.yml ]; then
  COMPOSE_CMD="docker compose -f docker-compose.test.yml"
else
  COMPOSE_CMD="docker compose"
fi

$COMPOSE_CMD up -d --build --wait
$COMPOSE_CMD ps

4c–4g. Fixtures, smoke tests, auth E2E, CRUD E2E, running

See references/e2e-patterns.md for complete templates:

• API helper + real auth fixture
• Docker Compose E2E override file
• Smoke / health tests
• Real auth browser flows
• Real CRUD tests (browser → API → DB assertion)
• How to start the full stack and run in order

---

When writing new tests

Integration test — Python example (real DB, no mocks):

import pytest
import httpx

@pytest.mark.asyncio
async def test_create_agent_persists(async_client: httpx.AsyncClient, db_session):
    # Act — call real API against real test DB
    response = await async_client.post("/api/agents", json={"name": "test-agent"})
    assert response.status_code == 201
    agent_id = response.json()["id"]

    # Assert — verify it actually persisted in the real DB
    row = await db_session.execute("SELECT name FROM agents WHERE id = $1", agent_id)
    assert row["name"] == "test-agent"

Integration test — JS/TS example (real DB via supertest):

it("POST /agents persists to database", async () => {
  const res = await request(app).post("/agents").send({ name: "test-agent" })
  expect(res.status).toBe(201)

  // Assert real DB state — not a mock return value
  const row = await db.query("SELECT name FROM agents WHERE id = $1", [res.body.id])
  expect(row.rows[0].name).toBe("test-agent")
})

Unit test — mock only external APIs:

# ✅ Correct: mock only the external payment API
async def test_create_subscription(monkeypatch):
    monkeypatch.setattr("stripe.Subscription.create", AsyncMock(return_value={"id": "sub_test"}))
    result = await billing_service.create_subscription(user_id=1, plan="pro")
    assert result.stripe_subscription_id == "sub_test"

# ❌ Wrong: don't mock the DB
async def test_create_subscription(monkeypatch):
    monkeypatch.setattr("db.session.add", MagicMock())  # Never do this

---

Output at each coverage iteration

=== Test Iteration N ===
Frontend:  lines X% | functions X% | branches X%
Backend:   lines X% | functions X% | branches X%
Python:    lines X% | functions X% | branches X%
Failing:   N tests
Action:    [what's being written / fixed]

Final output

=== /test Complete ===
Docker env:  ✅ services running | skipped (not needed)
Unit:        ✅ X passed
Integration: ✅ X passed (real DB)
Frontend:    lines ✅ X% | functions ✅ X% | branches ✅ X%
Backend:     lines ✅ X% | functions ✅ X% | branches ✅ X%
E2E:         ✅ X passed (full-stack Docker) | ⚠️ X failed (non-blocking) | skipped
Failures:    0 ✅
New files:   [list]
Status:      COVERAGE MET ✅