Multi-Cluster API Data Mismatch
Problem
When investigating why an API returns data that doesn't match direct database queries,
the root cause may be that your kubectl context points to a different cluster (e.g., POC)
than the one actually serving the public API (e.g., production). Both clusters have the
same schemas and similar data volumes, making the mismatch non-obvious.
Context / Trigger Conditions
- API returns records that direct database queries (
kubectl exec ... clickhouse-client)
can't find
- Data counts are similar (e.g., both have ~20k records) but specific records differ
- You're debugging a data pipeline that "should work" but labels/counts never update
- Multiple kubectl contexts exist (production, staging, POC, test)
- The API response header shows
server: nginx but no ingress exists in the current
cluster's namespace
Solution
Step 1: Verify which cluster serves the domain
# Check DNS for the public API domain
dig relay.divine.video +short
# → 34.58.27.79
# Check your current cluster's gateway/ingress IP
kubectl get gateway -A # or kubectl get ingress -A
# → 35.184.10.63 (different IP = different cluster!)
Step 2: Check all available contexts
kubectl config get-contexts
# Look for production vs staging vs POC contexts
Step 3: Switch to the correct cluster
kubectl config use-context <production-context-name>
Step 4: Verify the database connection
# Check what database the API actually connects to
kubectl get secret -n <namespace> <db-credentials> -o jsonpath='{.data.DATABASE_URL}' | base64 -d
# Production might use managed services (ClickHouse Cloud, Cloud SQL)
# POC might use in-cluster pods
Step 5: Also check for ExternalSecrets when updating secrets
If secrets are managed by ExternalSecrets operator:
# Check if ExternalSecrets manages the secret
kubectl get externalsecret -n <namespace>
# If yes, update the SOURCE (e.g., GCP Secret Manager), not the k8s secret
gcloud secrets versions add <secret-name> --project=<project> --data-file=-
# Force sync after updating
kubectl annotate externalsecret <name> -n <namespace> force-sync=$(date +%s) --overwrite
# Verify the k8s secret updated
kubectl get secret <name> -n <namespace> -o jsonpath='{.data.<key>}' | base64 -d
Verification
After switching to the correct cluster:
- Re-run the same database query — the "missing" records should now appear
- The latest timestamp in the database should match recent API activity
kubectl get pods -n <namespace> should show the same pods the API logs reference
Key Indicators You're On the Wrong Cluster
| Symptom | Explanation |
|---|
| API video count ~21k, DB count ~20k | Similar but not identical = different datasets |
| Latest DB record is days old | API is receiving writes on a different cluster |
| No ingress/gateway routes match the public domain | Traffic routes elsewhere |
| Managed DB URL (e.g., ClickHouse Cloud) vs in-cluster pod | Production vs POC architecture |
| API pod logs show only health checks, no real traffic | Real traffic goes to another cluster |
Example
# You query POC ClickHouse and find 20,371 videos, latest from Feb 26
# But the API at relay.divine.video returns videos from today
# The API returns a video with d_tag that doesn't exist in your ClickHouse
# Fix: switch to production
kubectl config use-context connectgateway_dv-platform-prod_...
# Now query production ClickHouse Cloud
CH_URL=$(kubectl get secret -n funnelcake funnelcake-clickhouse-credentials \
-o jsonpath='{.data.CLICKHOUSE_URL}' | base64 -d)
# → https://z1hzyismdt.us-central1.gcp.clickhouse.cloud:8443
# (managed service, not in-cluster pod)
# Query returns 21,221 videos with latest from minutes ago — matches the API
Notes
- Production often uses managed database services (ClickHouse Cloud, Cloud SQL)
while POC/staging use in-cluster pods — same schemas, different data
- The similar-but-not-identical record counts are the most misleading symptom
- Always check
max(created_at) or max(indexed_at) — if it's days old on a
live service, you're querying the wrong instance
- When updating k8s secrets in production, check for ExternalSecrets first —
manual
kubectl create secret will be immediately overwritten by the operator
syncing from GCP Secret Manager / AWS Secrets Manager / Vault