FreePublished on GitHub

funnelcake-deployment-workflow

Deploy funnelcake (api + relay) to ANY environment (production, staging, poc) on GKE via ArgoCD. Use when: (1) Deploying new funnelcake commits, (2) Building Docker images for GKE (amd64), (3) Running ClickHouse migrations, (4) Troubleshooting ImagePullBackOff errors, (5) Syncing staging/poc after production deployment. Covers complete workflow with PRE-FLIGHT CHECKLIST to prevent common deployment failures.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile

Works withDocker Kubernetes

See how to install

Files included (1)

SKILL.md13.9 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

Funnelcake Deployment Workflow

Problem

Deploying funnelcake requires multiple coordinated steps across DIFFERENT environments, each with its own container registry and ArgoCD instance. Common failures:

Updating kustomization with image tags that don't exist in target registry
Assuming push to main auto-deploys to staging/poc (it doesn't)
Not verifying pods actually started (vs stuck in ImagePullBackOff)

Context / Trigger Conditions

User asks to deploy funnelcake to any environment
New commits need to be deployed
Migrations need to be run
Pods stuck in ImagePullBackOff or ErrImagePull
Staging/poc is behind production

CRITICAL: Multi-Environment Architecture

Each environment is ISOLATED - nothing is shared!

Environment	Container Registry	ArgoCD	kubectl Context
Production	`us-central1-docker.pkg.dev/dv-platform-prod/containers-production/`	In production cluster	`connectgateway_dv-platform-prod_us-central1_gke-production-membership`
Staging	`us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/`	In staging cluster	`connectgateway_dv-platform-staging_us-central1_gke-staging-membership`
POC	`us-central1-docker.pkg.dev/rich-compiler-479518-d2/containers-poc/`	In POC cluster	`connectgateway_rich-compiler-479518-d2_us-central1_gke-poc-membership`

Key Facts:

Images in production registry are NOT available to staging/poc
Pushing to main branch does NOT auto-sync staging/poc ArgoCD
Each environment's ArgoCD must be synced separately

PRE-FLIGHT CHECKLIST (DO THIS FIRST!)

Before deploying to ANY environment, verify:

1. Does the image exist in the TARGET registry?

# Check what images exist
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/<PROJECT>/<REPO>/funnelcake-relay \
  --include-tags --limit=5

# Projects/repos by environment:
# - Production: dv-platform-prod/containers-production
# - Staging: dv-platform-staging/containers-staging
# - POC: rich-compiler-479518-d2/containers-poc

If image doesn't exist → BUILD AND PUSH IT FIRST!

2. Am I updating the correct overlay?

# Staging overlay:
k8s/applications/funnelcake-relay/overlays/staging/kustomization.yaml

# POC overlay:
k8s/applications/funnelcake-relay/overlays/poc/kustomization.yaml

# Production overlay:
k8s/applications/funnelcake-relay/overlays/production/kustomization.yaml

3. What's currently deployed vs what's in git?

# Check deployed image
kubectl --context <CONTEXT> get deployment funnelcake-relay -n funnelcake \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# Check git kustomization
cat k8s/applications/funnelcake-relay/overlays/<ENV>/kustomization.yaml | grep newTag

Complete Deployment Workflow

Step 1: Get Latest Code

cd /Users/rabble/code/divine/divine-funnelcake
git pull
git log --oneline -3  # Note the commit hash (e.g., e02d3b1)

Step 2: Build Images for AMD64 (CRITICAL)

IMPORTANT: Mac builds arm64 by default. GKE runs amd64. You MUST specify --platform linux/amd64.

# Build API image
docker buildx build --platform linux/amd64 --target api \
  -t us-central1-docker.pkg.dev/dv-platform-prod/containers-production/funnelcake-api:COMMIT_HASH \
  --push .

# Build Relay image
docker buildx build --platform linux/amd64 --target relay \
  -t us-central1-docker.pkg.dev/dv-platform-prod/containers-production/funnelcake-relay:COMMIT_HASH \
  --push .

If you get auth errors, run:

gcloud auth configure-docker us-central1-docker.pkg.dev --quiet

Step 3: Update Kustomize and Push

cd /Users/rabble/code/divine/divine-iac-coreconfig

# Update image tags
sed -i '' 's/newTag: "OLD_HASH"/newTag: "NEW_HASH"/' \
  k8s/applications/funnelcake-api/overlays/production/kustomization.yaml
sed -i '' 's/newTag: "OLD_HASH"/newTag: "NEW_HASH"/' \
  k8s/applications/funnelcake-relay/overlays/production/kustomization.yaml

# Commit and push (handle potential conflicts)
git add k8s/applications/funnelcake-*/overlays/production/kustomization.yaml
git commit -m "deploy(production): funnelcake NEW_HASH - description"
git pull --rebase origin main && git push origin main

Step 4: Trigger ArgoCD Sync

# Refresh to pick up git changes
kubectl patch application funnelcake-api -n argocd --type=merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'
kubectl patch application funnelcake-relay -n argocd --type=merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

# Wait, then trigger sync
sleep 3
kubectl patch application funnelcake-api -n argocd --type=merge \
  -p '{"operation":{"sync":{"revision":"HEAD"}}}'
kubectl patch application funnelcake-relay -n argocd --type=merge \
  -p '{"operation":{"sync":{"revision":"HEAD"}}}'

Step 5: Wait for Rollout

kubectl rollout status deploy/funnelcake-api deploy/funnelcake-relay \
  -n funnelcake --timeout=120s

Step 6: Verify

kubectl get deploy funnelcake-api -n funnelcake \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

Running Migrations

CRITICAL: Migration Entrypoint Has Two Modes

The database/entrypoint.sh detects ClickHouse connection mode from env vars:

Mode 1 (CLICKHOUSE_URL): Assumes ClickHouse Cloud — forces port 9440 with TLS. Use for production (ClickHouse Cloud at *.clickhouse.cloud).
Mode 2 (CLICKHOUSE_HOST + CLICKHOUSE_PORT): Direct host/port, no TLS assumption. Use for staging/poc (self-hosted ClickHouse on port 9000).

If you pass CLICKHOUSE_URL for self-hosted ClickHouse, migrations will fail with i/o timeout on port 9440 because the self-hosted instance listens on port 9000.

Build and Push Migration Image

cd /Users/rabble/code/divine/divine-funnelcake/database
docker buildx build --platform linux/amd64 \
  -t us-central1-docker.pkg.dev/dv-platform-prod/containers-production/funnelcake-migrate:COMMIT_HASH \
  --push .

Run Migration Job — PRODUCTION (ClickHouse Cloud)

kubectl delete job funnelcake-db-migrate -n funnelcake --ignore-not-found
cat <<'EOF' | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: funnelcake-db-migrate
  namespace: funnelcake
spec:
  ttlSecondsAfterFinished: 600
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: us-central1-docker.pkg.dev/dv-platform-prod/containers-production/funnelcake-migrate:COMMIT_HASH
          args: ["up"]
          env:
            - name: CLICKHOUSE_URL
              valueFrom:
                secretKeyRef:
                  name: funnelcake-clickhouse-credentials
                  key: CLICKHOUSE_URL
            - name: CLICKHOUSE_USER
              valueFrom:
                secretKeyRef:
                  name: funnelcake-clickhouse-credentials
                  key: CLICKHOUSE_USER
            - name: CLICKHOUSE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: funnelcake-clickhouse-credentials
                  key: CLICKHOUSE_PASSWORD
            - name: CLICKHOUSE_DATABASE
              valueFrom:
                secretKeyRef:
                  name: funnelcake-clickhouse-credentials
                  key: CLICKHOUSE_DATABASE
EOF

Run Migration Job — STAGING/POC (Self-hosted ClickHouse)

# Use CLICKHOUSE_HOST + CLICKHOUSE_PORT instead of CLICKHOUSE_URL!
kubectl --context <STAGING_CONTEXT> delete job funnelcake-db-migrate -n funnelcake --ignore-not-found
cat <<'EOF' | kubectl --context <STAGING_CONTEXT> apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: funnelcake-db-migrate
  namespace: funnelcake
spec:
  ttlSecondsAfterFinished: 600
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/funnelcake-migrate:COMMIT_HASH
          args: ["up"]
          env:
            - name: CLICKHOUSE_HOST
              value: "funnelcake-funnelcake-clickhouse.funnelcake.svc.cluster.local"
            - name: CLICKHOUSE_PORT
              value: "9000"
            - name: CLICKHOUSE_USER
              valueFrom:
                secretKeyRef:
                  name: funnelcake-clickhouse-credentials
                  key: CLICKHOUSE_USER
            - name: CLICKHOUSE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: funnelcake-clickhouse-credentials
                  key: CLICKHOUSE_PASSWORD
            - name: CLICKHOUSE_DATABASE
              valueFrom:
                secretKeyRef:
                  name: funnelcake-clickhouse-credentials
                  key: CLICKHOUSE_DATABASE
EOF

Check Migration Logs

sleep 10 && kubectl logs job/funnelcake-db-migrate -n funnelcake

Handle "Dirty Database" Error

If you see error: Dirty database version X, a previous migration failed partway:

# Force the version to clear dirty state
kubectl delete job funnelcake-db-migrate -n funnelcake --ignore-not-found
# Create job with args: ["force", "X"] instead of ["up"]
# Then run "up" again

Common Errors and Fixes

Error: "no match for platform in manifest: not found"

Cause: Image built for wrong architecture (arm64 on Mac, but GKE needs amd64) Fix: Rebuild with --platform linux/amd64

Error: "Unauthenticated request" when pushing

Cause: Docker not authenticated to Artifact Registry Fix: gcloud auth configure-docker us-central1-docker.pkg.dev --quiet

Error: "failed to push... fetch first"

Cause: Remote has new commits Fix: git pull --rebase origin main && git push origin main

Error: "Dirty database version X"

Cause: Previous migration failed midway Fix: Run force X to clear dirty state, then up again

Pods stuck in ImagePullBackOff

Causes:

Wrong architecture (check with kubectl describe pod)
Image doesn't exist (typo in tag)
Auth issues (check imagePullSecrets)

Key Paths

Funnelcake repo: /Users/rabble/code/divine/divine-funnelcake
IaC repo: /Users/rabble/code/divine/divine-iac-coreconfig
API kustomization: k8s/applications/funnelcake-api/overlays/production/kustomization.yaml
Relay kustomization: k8s/applications/funnelcake-relay/overlays/production/kustomization.yaml
Migrations: /Users/rabble/code/divine/divine-funnelcake/database/migrations/

Deploying to Staging or POC

Full Workflow for Non-Production Environments

# 1. Check what's in production that we want to deploy
cat k8s/applications/funnelcake-relay/overlays/production/kustomization.yaml | grep newTag
# e.g., newTag: "e7e79eb"

# 2. CHECK if that image exists in staging registry
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/funnelcake-relay \
  --include-tags --limit=10 | grep e7e79eb

# 3. If NOT found → BUILD AND PUSH
cd /Users/rabble/code/divine/divine-funnelcake
git checkout e7e79eb

docker build --platform linux/amd64 --target relay \
  -t us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/funnelcake-relay:e7e79eb .
docker push us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/funnelcake-relay:e7e79eb

docker build --platform linux/amd64 --target api \
  -t us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/funnelcake-api:e7e79eb .
docker push us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/funnelcake-api:e7e79eb

# For POC, tag and push to POC registry too
docker tag ...staging...:e7e79eb ...poc...:e7e79eb
docker push ...poc...:e7e79eb

git checkout main  # Return to main

# 4. Update kustomization (only AFTER images exist!)
cd /Users/rabble/code/divine/divine-iac-coreconfig
# Edit staging/poc overlays with new tags

# 5. Commit and push
git add -A && git commit -m "deploy(staging, poc): funnelcake @ e7e79eb" && git push

# 6. Sync ArgoCD (each environment has its own!)
kubectl --context connectgateway_dv-platform-staging_us-central1_gke-staging-membership \
  patch application funnelcake-relay -n argocd --type merge \
  -p '{"operation":{"initiatedBy":{"username":"claude"},"sync":{"syncStrategy":{"apply":{"force":false}}}}}'

# 7. VERIFY pods are actually running (not ImagePullBackOff!)
kubectl --context connectgateway_dv-platform-staging_us-central1_gke-staging-membership \
  get pods -n funnelcake

POST-DEPLOYMENT VERIFICATION CHECKLIST

Always run these after any deployment:

# 1. Check pod status (should be Running, not ImagePullBackOff)
kubectl --context <CONTEXT> get pods -n funnelcake

# 2. Check actual deployed image matches expected
kubectl --context <CONTEXT> get deployment funnelcake-relay -n funnelcake \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# 3. Check the website actually works
curl -s https://relay.staging.dvines.org/ | grep -o "version.*" | head -1

Image Registry

Production: us-central1-docker.pkg.dev/dv-platform-prod/containers-production/
Staging: us-central1-docker.pkg.dev/dv-platform-staging/containers-staging/
POC: us-central1-docker.pkg.dev/rich-compiler-479518-d2/containers-poc/

Notes

Always use short commit hash (7 chars) for image tags
Dockerfile has multi-stage build: --target api or --target relay
Migration image is in database/Dockerfile
ClickHouse credentials are in funnelcake-clickhouse-credentials secret
ArgoCD needs BOTH refresh AND sync operations

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

More skills from this repository

argocd-externalsecret-namespace-permission

Free

Fix ArgoCD ExternalSecret deployment failing with "namespace X is not permitted in project Y". Use when: (1) ExternalSecret shows OutOfSync in ArgoCD but won't sync, (2) ArgoCD application status shows "namespace X is not permitted in project 'infrastructure'", (3) ExternalSecret targets a namespace managed by a different ArgoCD project, (4) Using apps-of-apps pattern with separate infrastructure and application projects.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

art-direct

Free

Art direction for any content — reads text, PDF, Word, HTML, PPT, then proposes 2-3 creative directions with photography style, mood, and visual language. After selection, generates AI image prompts and visual briefs section-by-section. Use when the user shares content and needs visual direction, image sourcing, or creative direction for any material.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

async-await-null-race-condition

Free

Fix "Null check operator used on a null value" errors when an object is set to null during an async await. Use when: (1) Object reference is nullified while awaiting, (2) Code accesses object with ! after await returns, (3) Cancel/dispose operations run concurrently with async operations on same object. Solution: capture local reference before await.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

aws-v4-signing-custom-headers-gcs

Free

Add custom metadata headers (x-amz-meta-*) to AWS v4 signed requests for GCS S3-compatible API. Use when: (1) Adding custom metadata to GCS uploads via S3 API, (2) Getting signature mismatch errors after adding new headers, (3) x-amz-meta-* headers being ignored or causing 403 errors. Custom headers MUST be included in canonical headers and signed headers list.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

bash-herestring-newline-secrets

Free

Fix password/secret authentication failures caused by trailing newlines when creating Google Cloud secrets (or similar) with bash here-strings. Use when: (1) Password authentication fails with correct password, (2) Secret created with `<<< "value"` syntax, (3) Error like "password authentication failed" or "invalid token" despite correct value. Bash here-strings (`<<<`) add a trailing newline that corrupts secrets.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

blossom-content-addressed-url-filtering

Free

Fix silent video/media processing failures caused by URL extraction code that filters on file extensions (.mp4, .webm, .webp). Use when: (1) Media moderation, transcoding, or analysis silently skips files from Blossom or content-addressed storage servers, (2) URL extraction from Nostr event tags (imeta, r tags) drops URLs without recognized extensions, (3) CDN fallback URLs append .mp4 but the actual server uses extensionless content-addressed paths like /{sha256}. Common in Nostr video events (kind 34236) where different clients use different URL formats.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

See all skills by divinevideo

Report a problem with this skill

Install

Install with a command

npx skills add https://github.com/divinevideo/divine-mobile --skill funnelcake-deployment-workflow

Paste it into your terminal and run it (requires Node.js). You can choose which agents to install it for, such as Claude Code or Codex.

Ask your agent

Please install the following Agent Skill.
https://github.com/divinevideo/divine-mobile/tree/main/.claude/skills/funnelcake-deployment-workflow

First review SKILL.md and the bundled files and confirm there is nothing suspicious (such as sending data externally or reading credentials). Then place it in ~/.claude/skills/funnelcake-deployment-workflow/ for Claude Code, or ~/.agents/skills/funnelcake-deployment-workflow/ for Codex.

Send this text to Claude Code or Codex as is. It checks the contents before installing.

View source on GitHub

Repository: divinevideo/divine-mobile
Stars: 266
Forks: 56
Category: DevOps
License: MPL-2.0
Original language: English
Last updated: Oct 19, 2026

A skill is a set of instructions and scripts for an agent. cccskills does not guarantee that skills are safe. Review the contents before you install.