Cloud Trace Skill

Manage and analyze Google Cloud Trace using gcloud trace and monitoring commands.

Discovery-First Rule

ALWAYS discover before acting. Never assume trace IDs, span names, or service names. Discover available traces and services first.

# Discover recent traces
gcloud trace traces list --limit=20 --format=json \
  | jq '[.[] | {traceId: .traceId, projectId: .projectId, spans: [.spans[:3][] | {spanId: .spanId, name: .name, startTime: .startTime, endTime: .endTime}]}]'

Parallel Execution Requirement

ALL independent operations MUST run in parallel using background jobs (&) and wait.

for trace_id in $(gcloud trace traces list --limit=10 --format="value(traceId)"); do
  {
    gcloud trace traces describe "$trace_id" --format=json
  } &
done
wait

Helper Functions

# Get trace details
get_trace() {
  local trace_id="$1"
  gcloud trace traces describe "$trace_id" --format=json \
    | jq '{traceId: .traceId, spans: [.spans[] | {spanId: .spanId, name: .name, kind: .kind, startTime: .startTime, endTime: .endTime, status: .status, labels: .labels, parentSpanId: .parentSpanId}]}'
}

# List traces with filter
list_traces() {
  local filter="$1" limit="${2:-20}"
  gcloud trace traces list --filter="$filter" --limit="$limit" --format=json
}

# Get trace latency metrics via monitoring
get_latency_metrics() {
  local service="$1"
  gcloud monitoring time-series list \
    --filter="metric.type=\"cloudtrace.googleapis.com/http/server/response_latencies\" AND metric.labels.service=\"$service\"" \
    --interval-start-time="$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
    --format=json
}

# Get trace count metrics
get_trace_counts() {
  gcloud monitoring time-series list \
    --filter="metric.type=\"cloudtrace.googleapis.com/http/server/response_count\"" \
    --interval-start-time="$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
    --format=json
}

Common Operations

1. Latency Analysis

# Overall request latency distribution
gcloud monitoring time-series list \
  --filter="metric.type=\"cloudtrace.googleapis.com/http/server/response_latencies\"" \
  --interval-start-time="$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --format=json

# Latency by service
gcloud monitoring time-series list \
  --filter="metric.type=\"cloudtrace.googleapis.com/http/server/response_latencies\"" \
  --interval-start-time="$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --format=json \
  | jq '[.[] | {service: .metric.labels.service, method: .metric.labels.method, distributionValue: .points[0].value.distributionValue}]'

2. Trace Exploration

# Recent slow traces (sorted by duration)
gcloud trace traces list --limit=50 --format=json \
  | jq '[.[] | {traceId: .traceId, rootSpan: .spans[0].name, startTime: .spans[0].startTime, durationMs: ((.spans[0].endTime | sub("Z$";"") | split(".")[0] | strptime("%Y-%m-%dT%H:%M:%S") | mktime) - (.spans[0].startTime | sub("Z$";"") | split(".")[0] | strptime("%Y-%m-%dT%H:%M:%S") | mktime)) * 1000}] | sort_by(-.durationMs) | .[:10]'

# Trace with specific span name
list_traces "span:\"$SPAN_NAME\"" 20

# Error traces
list_traces "status.code!=0" 20

3. Span Analysis

# Get full span tree for a trace
get_trace "$TRACE_ID"

# Find slowest spans in a trace
gcloud trace traces describe "$TRACE_ID" --format=json \
  | jq '[.spans[] | {name: .name, kind: .kind, labels: .labels, parentSpanId: .parentSpanId}] | sort_by(-.durationMs) | .[:5]'

4. Sampling Configuration

# Check trace sampling via monitoring (sampled vs total)
gcloud monitoring time-series list \
  --filter="metric.type=\"cloudtrace.googleapis.com/http/server/response_count\"" \
  --interval-start-time="$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --format=json

# Trace configuration is typically set in application code or via Cloud Trace agent config
# Check if any oversampling settings exist
gcloud services list --filter="name:cloudtrace.googleapis.com" --format=json

5. Cross-Service Trace Correlation

# Find traces spanning multiple services
gcloud trace traces list --limit=20 --format=json \
  | jq '[.[] | {traceId: .traceId, services: [.spans[].labels."g.co/agent" // .spans[].labels.component // "unknown"] | unique, spanCount: (.spans | length)}] | [.[] | select(.services | length > 1)]'

# Get detailed view of a cross-service trace
get_trace "$TRACE_ID"

Output Format

Present results as a structured report:

Gcp Cloud Trace Report
══════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

1. Sampling rate: Cloud Trace samples approximately 0.1 requests per second per project by default. Low-traffic services may have very few traces. Force tracing with X-Cloud-Trace-Context header.
2. Trace retention: Traces are retained for 30 days. Older traces are automatically deleted and cannot be recovered.
3. Span limits: A single trace can have at most 128 spans. Large distributed transactions may be truncated.
4. Latency vs duration: Trace latency includes network time between services. Individual span duration is the time spent in that service only.
5. Missing spans: Missing child spans usually indicate the downstream service is not instrumented or sampling dropped the span. Check instrumentation on all services in the call chain.