Cloud Logging Skill

Manage and analyze Google Cloud Logging using gcloud logging commands.

Discovery-First Rule

ALWAYS discover before acting. Never assume log names, sink names, metric names, or exclusion filter names.

# Discover log names
gcloud logging logs list --format=json --limit=50 \
  | jq '[.[] | split("/") | last]'

# Discover sinks
gcloud logging sinks list --format=json \
  | jq '[.[] | {name: .name, destination: .destination, filter: .filter, disabled: .disabled}]'

Parallel Execution Requirement

ALL independent operations MUST run in parallel using background jobs (&) and wait.

for sink in $(gcloud logging sinks list --format="value(name)"); do
  {
    gcloud logging sinks describe "$sink" --format=json
  } &
done
wait

Helper Functions

# Read logs with filter
read_logs() {
  local filter="$1" limit="${2:-100}"
  gcloud logging read "$filter" --limit="$limit" --format=json \
    | jq '[.[] | {timestamp: .timestamp, severity: .severity, logName: .logName | split("/") | last, message: .textPayload // .jsonPayload.message // (.jsonPayload | tostring | .[:200])}]'
}

# List log-based metrics
list_log_metrics() {
  gcloud logging metrics list --format=json \
    | jq '[.[] | {name: .name, description: .description, filter: .filter, metricDescriptor: .metricDescriptor.type}]'
}

# Get sink details
get_sink_details() {
  local sink="$1"
  gcloud logging sinks describe "$sink" --format=json \
    | jq '{name: .name, destination: .destination, filter: .filter, disabled: .disabled, exclusions: .exclusions, writerIdentity: .writerIdentity, includeChildren: .includeChildren}'
}

# List exclusion filters
list_exclusions() {
  gcloud logging sinks list --format=json \
    | jq '[.[] | select(.exclusions) | {sink: .name, exclusions: .exclusions}]'
}

Common Operations

1. Log Analysis

# Recent errors across all services
read_logs "severity>=ERROR" 50

# Errors for a specific service
read_logs "resource.type=\"cloud_run_revision\" AND severity>=ERROR" 50

# Audit logs for IAM changes
read_logs "logName:\"activity\" AND protoPayload.methodName:\"SetIamPolicy\"" 20

# GKE container logs
read_logs "resource.type=\"k8s_container\" AND resource.labels.namespace_name=\"$NAMESPACE\"" 100

2. Log-Based Metrics

# List all custom metrics
list_log_metrics

# Get metric details
gcloud logging metrics describe "$METRIC_NAME" --format=json \
  | jq '{name: .name, filter: .filter, description: .description, labelExtractors: .labelExtractors, bucketOptions: .bucketOptions}'

# Check metric values via monitoring
gcloud monitoring time-series list \
  --filter="metric.type=\"logging.googleapis.com/user/$METRIC_NAME\"" \
  --interval-start-time="$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --format=json

3. Sink Management

# All sinks with destinations
gcloud logging sinks list --format=json \
  | jq '[.[] | {name: .name, destination: .destination, filter: .filter, disabled: .disabled, writerIdentity: .writerIdentity}]'

# Check sink writer identity permissions
for sink in $(gcloud logging sinks list --format="value(name)"); do
  {
    echo "Sink: $sink"
    get_sink_details "$sink"
  } &
done
wait

4. Exclusion Filter Analysis

# List all exclusion filters
gcloud logging sinks describe "_Default" --format=json \
  | jq '.exclusions // [] | [.[] | {name: .name, filter: .filter, disabled: .disabled, description: .description}]'

# Check what percentage of logs are excluded
gcloud monitoring time-series list \
  --filter="metric.type=\"logging.googleapis.com/exports/byte_count\"" \
  --interval-start-time="$(date -u -v-24H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --format=json

5. Log Volume and Cost Analysis

# Ingestion volume by log type
gcloud monitoring time-series list \
  --filter="metric.type=\"logging.googleapis.com/billing/bytes_ingested\"" \
  --interval-start-time="$(date -u -v-24H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --format=json

# Monthly ingested bytes per log name
gcloud monitoring time-series list \
  --filter="metric.type=\"logging.googleapis.com/billing/monthly_bytes_ingested\"" \
  --interval-start-time="$(date -u -v-24H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --format=json

Output Format

Present results as a structured report:

Gcp Cloud Logging Report
════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

1. _Required vs _Default sink: The _Required sink cannot be disabled or filtered (audit logs, access transparency). Only _Default and custom sinks can be configured.
2. Exclusion filter cost: Excluded logs are not stored but are still ingested. They count toward ingestion quotas but not storage costs.
3. Log filter syntax: Cloud Logging uses a filter expression language, not regex. Use =~ for regex matching: textPayload =~ "error.*timeout".
4. Sink permissions: Each sink has a writerIdentity service account that needs write permissions on the destination (BigQuery, GCS, Pub/Sub).
5. Retention periods: Default retention is 30 days for _Default bucket. Custom buckets can have different retention. Locked buckets cannot be shortened.