Centaur QA

Run this skill from a new Slack thread in a channel where the bot is present. The goal is to prove the user-facing agent path works, not just that APIs respond.

Default behavior: start immediately, run the in-thread smoke test, and return a concise pass/fail report. Do not ask clarifying questions unless the current Slack channel or thread cannot be inferred.

If the user asks for deploy readiness, promotion gating, concurrency checks, or broad tool coverage, run the in-thread smoke test first, then the relevant extended checks below.

Success Criteria

The smoke test passes only when all core workflows work from the running agent session:

• A list of tools loads and includes expected tools.
• Slack file upload to the current thread works.
• Slack file download from the current thread works.
• Slack file download from the current channel, then re-upload to the current thread, works.
• Slack token search works after bounded retries.
• Slack overall message search works.
• Current thread history works.
• Company context connects to its database and returns indexed documents or a valid empty result.
• VictoriaLogs and VictoriaMetrics are reachable, and VictoriaMetrics proves metric existence without asserting exact metric values.

Treat auth, permission, DNS, schema, and timeout errors as failures. Treat empty search results as warnings only when the tool successfully queried the backing service.

For promotion or deploy-readiness requests, the final status is PASS only if the requested extended checks also pass or are explicitly accepted by the owner.

In-Thread Smoke Test

Use direct tool CLIs when available. Use centaur-tools call <tool> <method> '<json>' when a tool has no standalone CLI command for the method you need.

Slack file transfers must use the proxied commands, which route through the Centaur API server Slack proxy: slack upload and slack download <file_id> <channel_id>. Do not use upload-direct, download-direct, slack files --download, centaur-tools call slack upload_file, or raw url_private fetches; those bypass the proxy and do not prove the user-facing path.

First capture session context:

echo "THREAD_KEY=${CENTAUR_THREAD_KEY:-}"
echo "SLACK_CHANNEL_ID=${SLACK_CHANNEL_ID:-}"
echo "SLACK_THREAD_TS=${SLACK_THREAD_TS:-}"
echo "SLACK_CHANNEL_NAME=${SLACK_CHANNEL_NAME:-}"

If SLACK_CHANNEL_ID or SLACK_THREAD_TS is missing, infer it from CENTAUR_THREAD_KEY when possible. Slack thread keys are usually slack:<channel_id>:<thread_ts>.

1. Tool Loading

centaur-tools list

Verify that the list is non-empty and includes at least slack, company_context, vlogs, and vmetrics. If a tool is absent, record the failure before continuing.

2. Upload File To Current Thread

Upload a tiny deterministic file to the current Slack thread through the proxy:

QA_TOKEN="centaur-qa-$(date +%s)"
QA_UPLOAD_DIR="/tmp/centaur-qa-upload-${QA_TOKEN}"
mkdir -p "$QA_UPLOAD_DIR"
printf '%s\n' "$QA_TOKEN" > "$QA_UPLOAD_DIR/centaur-qa-upload.txt"
slack upload "${SLACK_CHANNEL_ID}" "$QA_UPLOAD_DIR/centaur-qa-upload.txt" \
  --thread "${SLACK_THREAD_TS}" \
  --comment "QA upload smoke test: ${QA_TOKEN}"

Verify the command prints Uploaded centaur-qa-upload.txt and a Slack file ID (F...). Save the file ID for the next step. If no file ID is printed, use thread history in step 3 to find the uploaded file's id.

3. Download File From Current Thread

If step 2 did not print a file ID, fetch the current thread history and take the uploaded file's id:

slack thread "${SLACK_CHANNEL_ID}:${SLACK_THREAD_TS}" --json --limit 20

Download it through the proxy to a sandbox-local temp directory:

QA_DOWNLOAD_DIR="/tmp/centaur-qa-files-${QA_TOKEN}"
slack download "${FILE_ID_FROM_CURRENT_THREAD}" "${SLACK_CHANNEL_ID}" --output "$QA_DOWNLOAD_DIR"

Verify the JSON output includes output_path and a non-zero size_bytes, and that the downloaded file contains $QA_TOKEN.

4. Current Thread History

slack thread "${SLACK_CHANNEL_ID}:${SLACK_THREAD_TS}" --json --limit 20

Verify the returned messages include the QA request and the file upload message. Record the number of messages returned.

5. Search Uploaded Token

Search for the unique token uploaded in step 2:

for attempt in 1 2 3; do
  slack search "$QA_TOKEN" --limit 5 --full && break
  sleep 10
done

Verify at least one result points to the current channel or current thread. Retry up to three total attempts with 10 seconds between attempts to handle Slack indexing lag. If all three attempts complete successfully but return no matching result, record WARN: token not indexed yet and continue.

6. Search Overall Messages

Run a separate broader message search for a stable term from the current channel:

slack search "${SLACK_CHANNEL_NAME:-centaur}" --limit 5 --full

Pass when the command succeeds and returns valid search output. Empty results are a warning only when the query reached Slack successfully.

7. Download File From Current Channel And Re-Upload

Find another accessible Slack file from the current channel. Prefer a result outside the current thread, but do not use files from other channels. search_files filters by filename or title, so start broad with an empty query:

slack search-files "${SLACK_CHANNEL_ID}" "" --limit 5

Pick a result whose channels includes ${SLACK_CHANNEL_ID}. If possible, avoid the file uploaded earlier in this QA run so the check proves download-and-reupload of an existing channel file. Download it through the proxy by file ID:

QA_REUPLOAD_DIR="/tmp/centaur-qa-reupload-${QA_TOKEN}"
slack download "${FILE_ID_FROM_CURRENT_CHANNEL}" "${SLACK_CHANNEL_ID}" --output "$QA_REUPLOAD_DIR"

Save the output_path from the JSON output, then re-upload it to the current thread through the proxy:

slack upload "${SLACK_CHANNEL_ID}" "${DOWNLOADED_FILE_FROM_CURRENT_CHANNEL}" \
  --thread "${SLACK_THREAD_TS}" \
  --comment "QA re-upload from current channel"

Verify the current thread now shows the re-uploaded file. If no accessible current-channel file exists, record SKIP: no readable file in current channel and include the search-files result.

8. Company Context

This verifies the database-backed context path and row-level permissions:

company_context list --limit 3 --json
company_context search "centaur" --limit 3 --json

Pass when the tool returns a valid JSON payload with status: ok, even if no documents match. Fail on database connection errors, permission errors, missing COMPANY_CONTEXT_DSN, or malformed results.

If company context returns upstream connection failed, use runtime evidence before suggesting a code fix:

centaur-tools call vlogs thread_logs '{"thread_key":"'"${CENTAUR_THREAD_KEY}"'","start":"2h"}'
vlogs query '_time:2h ("upstream connection failed" OR "postgres upstream connect failed" OR "failed to fetch secret")' --limit 20 --json

Classify whether the failure is a database proxy upstream issue, missing database selection, secret-resolution failure, or tool schema/client error. Include the failing tool command and thread key in the report.

9. Logs Via VictoriaLogs

vlogs health
vlogs query '*' --limit 3 --json
vlogs query "centaur.thread_key:\"${CENTAUR_THREAD_KEY}\"" --limit 10 --json

Pass when VictoriaLogs is reachable and returns JSON log entries or a valid empty list for the current-thread query. Fail on DNS, HTTP, or LogsQL errors. Keep this check scoped to the current deployment state visible from the current thread.

Do not run broad error searches such as level:error, vlogs errors, or error OR ERROR as part of the default QA pass/fail decision. The goal of this step is to prove the vlogs connection works, not to audit whether unrelated services currently have errors. Only query error logs when a preceding check failed and you need runtime evidence to classify that specific failure.

10. Metrics Via VictoriaMetrics

vmetrics health
vmetrics query 'count(up)' --json
vmetrics series 'up' --limit 5 --json

Pass when VictoriaMetrics is reachable and the responses prove that at least one up series exists. Do not use catch-all selectors such as {__name__=~".+"}; they scan every series and are too expensive on shared environments. Do not assert exact sample values, counter values, gauge values, label values, or a specific up value; those are environment-dependent. Fail on DNS, HTTP, malformed response errors, or a valid response that proves no metric names or series exist.

Extended Checks

Run these after the core smoke test when the user asks for deploy readiness, concurrency, broad tool coverage, or promotion confidence.

Tool Health

For broad tool coverage, follow the tool-health-smoke skill. It runs <tool> health for every live tool CLI through the normal brokered-credential path. Use health as the canonical tool smoke surface; only run other read-only commands after recording a failed health result, to classify that failure.

Concurrent Agent Turns

When asked to check concurrency or deadlocks, start 3-5 QA prompts in separate Slack threads in the same channel. Use distinct QA_TOKEN values and ask each agent to do a different read-only task:

• Read thread history and summarize earlier messages.
• Call two tools and summarize grounded results.
• Query logs for its own thread key.
• Search messages for a synthetic token.
• Use company context for a small internal DB lookup.

Pass when every turn reaches a terminal response in Slack, no thread remains stuck busy, and vlogs show one coherent execution per prompt without duplicate final delivery.

Promotion Gate

A deployment is ready for promotion only when:

• The core in-thread smoke test passes.
• Requested extended checks pass or failures are explicitly accepted by the owner.
• The same commit/build was tested and is the one being promoted.
• The report includes enough evidence for another engineer to verify: Slack permalinks, thread key, execution IDs, and log query windows.

Report Format

Reply in the Slack thread with a digestible QA report, not a prose paragraph. The first line must answer the outcome:

Overall: PASS|FAIL|PARTIAL - <one short reason>

Use PASS only when every required smoke check passed. Use PARTIAL when the user-facing path mostly worked but at least one required check warned, skipped, or could not be verified. Use FAIL when any required check hit an auth, permission, DNS, schema, timeout, malformed-response, upload/download, or backing-service error.

Then include a Slack-friendly digest. Do not use Markdown tables in Slack responses; Slack does not render them reliably. Use this exact shape:

*Setup*
- *Thread context:* PASS - C123:1712345678.000000, key slack:C123:...
- *Tool loading:* PASS - 72 tools; expected slack/company_context/vlogs/vmetrics present

*Slack*
- *Upload current-thread file:* PASS - F123, centaur-qa-upload.txt
- *Download current-thread file:* PASS - centaur-qa-upload.txt, 22 bytes, token matched
- *Re-upload current-channel file:* SKIP - no readable prior file found in channel
- *Search uploaded token:* WARN - 3 attempts, 0 results; likely indexing lag
- *Search overall messages:* PASS - query "centaur", 5 results
- *Current thread history:* PASS - 4 messages; QA request and upload present

*Data + Observability*
- *Company context:* PASS - list 0, search 0, status ok
- *VictoriaLogs:* PASS - health ok, sample 3, thread query 0
- *VictoriaMetrics:* PASS - reachable, `up` series found

*Extended*
- *Tool health:* SKIP - not requested
- *Concurrency:* SKIP - not requested

Rules for the digest:

• Keep each evidence phrase to one short sentence fragment. Prefer concrete IDs, counts, filenames, byte counts, query names, and attempt counts.
• Do not write mostly PASS. Use PARTIAL and put each warning or skipped item on its own line.
• Do not paste raw JSON, stack traces, credentials, tokens, or long command output. Summarize the failure class and the command or endpoint that failed.
• If Slack file upload creates visible artifacts, mention the uploaded filenames and file IDs so the user can correlate them with thread attachments.
• If a check is skipped because no suitable input exists, mark only that row SKIP; do not hide it in prose.
• If using the Slack API directly, a Block Kit version is acceptable only when the message still contains the same information: headline, grouped sections, per-check result, and evidence. Do not require Block Kit for normal assistant replies.

After the grouped digest, add at most three short bullets. Use Slack mrkdwn list items with - prefixes:

• Failures: highest-signal failed rows and likely owner.
• Warnings: non-blocking warnings such as Slack indexing lag or empty-but-valid search/log results.
• Promotion: ready, not ready, or not evaluated, with one reason.

Omit any bullet that has no content. The whole report should fit comfortably in one Slack message.

Known Gotchas

Symptom	Likely Cause	Action
Missing Slack env vars	Invocation did not come through Slack, or runtime metadata was not injected	Derive from CENTAUR_THREAD_KEY; otherwise fail early
slack download rejects arguments	Passed a URL or permalink instead of a file ID, or the channel is not where the file is shared	Read thread history or search-files JSON and pass the file id (F...) plus the channel ID it is shared in
Search cannot find just-uploaded token	Slack search indexing lag	Retry up to three total attempts, then warn and run the separate overall message search
company_context permission denied	Principal lacks DB-backed reader grant	Report principal/channel and ask owner to grant company context access
company_context upstream connection failed	Database proxy upstream, database selection, or secret-resolution failure	Check thread logs and vlogs for upstream connect and secret fetch errors before proposing a code change
Tool CLI import error	Package entrypoint or relative import packaging regression	Run <tool> health, <tool> --help, package build checks, and report the broken console script
Tool crashes when an API key env var is absent	Client assumes local env auth despite brokered credentials	Re-run <tool> health with the env var unset and verify client construction does not raise
Tool returns 401/403	Missing grant, wrong secret, or header not injected by iron-proxy	Check the principal's grants and the tool's [tool.centaur] hosts/match rules, then follow the auth-failure-log-triage skill
vlogs or vmetrics DNS failure	Observability service unavailable from sandbox	Check local stack or cluster service deployment
Expected tools missing	Tool catalog did not load or overlay masked base tools	Report the missing tool names and include centaur-tools list output
Concurrent runs hang	Runtime assignment, execution queue, or final delivery issue	Check session_executions, session_events, and the vlogs thread trace

Failure Triage

When a flow fails, inspect runtime evidence before redesigning:

• Stuck execution: check session_executions, session_events, and centaur-tools call vlogs thread_trace '{"thread_key":"..."}'.
• Missing Slack response: check Slackbot logs, final delivery state, and the Slack thread surface.
• File failure: check slack file-info <file_id> <channel_id>, downloaded byte size, and the proxied upload response.
• Tool failure: classify credential, DNS, upstream, schema, timeout, or runtime errors separately.
• Brokered-auth failure: verify whether the tool should work without local env vars and whether the proxy injected the expected headers.
• Context bug: inspect thread history, requester context, and message ordering.

References

Reference	When To Read
../tool-health-smoke/SKILL.md	Broad all-tool health coverage
../auth-failure-log-triage/SKILL.md	A tool fails with an auth or permission error
references/test-inputs.md	Only for deeper read-only triage after a tool health check fails