本文へ移動
cccskills
無料GitHub で公開

gke-ai-troubleshooting-tpu-metrics-monitoring

Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.

インストール方法を見る

含まれるファイル(3)

  • SKILL.md7.3 KB
  • references/failure_signatures.md1.4 KB
  • scripts/validate_queries.sh348 B

SKILL.md(原文)

インストールする前に、エージェントに与えられる指示の中身を確認できます。

GKE TPU Metrics Monitoring Guide

This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure.

Step 0: Mandatory Context

Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided {variable} placeholders:

  • {project_id}: The GCP Project ID.
  • {cluster_name}: The GKE Cluster Name.
  • {location}: The GKE Cluster Location (region or zone).
  • {node_name}: (Optional) The name of the specific GKE node.
  • {node_pool_name}: (Optional) The name of the GKE node pool.

Diagnostic Steps

Step 1: Verify TPU Runtime Metrics Configuration [Low Risk] [Auto]

Before analyzing runtime metrics, verify that the workload is configured to export them. This ensures the cluster and container environment are set up for automated metric scraping and visibility into accelerator health.

  • Action: Verify that the Pod specification and cluster meet the following prerequisites:
    • containerPort: 8431 exposed on the TPU container (required for Prometheus metric scraping).
    • JAX version 0.4.14 or later if using JAX (earlier versions do not export runtime metrics).
    • GKE version is 1.27.4-gke.900 or later (required for TPU runtime metric support).
    • GKE System Metrics are enabled on the cluster (required for Cloud Monitoring ingestion).

Step 2: Monitor TPU Runtime Metrics [Low Risk] [Auto]

If configured correctly, the following metrics are available in Cloud Monitoring (monitored resources k8s_node and k8s_container):

  • Container Metrics:
    • kubernetes.io/container/accelerator/duty_cycle: Percentage of time over the past sampling period (60 seconds) during which the TensorCores were actively processing on a TPU chip.
    • kubernetes.io/container/accelerator/memory_used: Amount of accelerator memory allocated in bytes.
    • kubernetes.io/container/accelerator/memory_total: Total accelerator memory in bytes.
  • Node Metrics:
    • kubernetes.io/node/accelerator/duty_cycle
    • kubernetes.io/node/accelerator/memory_used
    • kubernetes.io/node/accelerator/memory_total

Step 3: Check Node Status Condition [Low Risk] [Auto]

Query the status condition of GKE nodes (GKE version 1.32.1-gke.1357001 or later).

  • PromQL Query (Check if a specific node is Ready):
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"}
    
  • PromQL Query (List nodes with non-Ready conditions that are True):
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"}
    
  • PromQL Query (List nodes that are NOT Ready):
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}
    
  • PromQL Query (Fleet-wide node status):
    avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m]))
    

Step 4: Check Node Pool Status [Low Risk] [Auto]

Query the status of multi-host TPU node pools.

  • PromQL Query (Verify if a specific node pool is Running):
    kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"}
    
  • PromQL Query (Monitor node pools grouped by status):
    count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m]))
    
    Possible statuses: Provisioning, Running, Error, Reconciling, Stopping.

Step 5: Check Node Pool Availability [Low Risk] [Auto]

Query if all nodes in a multi-host TPU node pool are available.

  • PromQL Query (Check availability over time):
    avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m]))
    
    Value: 1 (True, all nodes available) or 0 (False, some nodes unavailable).

Step 6: Analyze Node Interruptions [Low Risk] [Auto]

Query the count of interruptions for GKE nodes.

  • PromQL Query (Breakdown of interruptions and causes):
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m]))
    
    Interruption Types: TerminationEvent, MaintenanceEvent, PreemptionEvent. Interruption Reasons: HostError, Eviction, AutoRepair.
  • PromQL Query (Filter for Host Maintenance events):
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m]))
    
  • PromQL Query (Interruption count aggregated by node pool):
    sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m]))
    

Step 7: Calculate Recovery and Interruption Metrics [Low Risk] [Auto]

Calculate Mean Time to Recovery (MTTR) and Mean Time Between Interruptions (MTBI) over the last 7 days.

  • PromQL Query (MTTR - Mean Time to Recovery):
    sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d]))
    
  • PromQL Query (MTBI - Mean Time Between Interruptions):
    sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d]))
    

Step 8: Monitor TPU Host Metrics [Low Risk] [Auto]

For GKE version 1.28.1-gke.1066000 or later, monitor TPU host performance.

  • Container Metrics:
    • kubernetes.io/container/accelerator/tensorcore_utilization: Current percentage of the TensorCore that is utilized.
    • kubernetes.io/container/accelerator/memory_bandwidth_utilization: Current percentage of the accelerator memory bandwidth that is being used.
  • Node Metrics:
    • kubernetes.io/node/accelerator/tensorcore_utilization
    • kubernetes.io/node/accelerator/memory_bandwidth_utilization

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

同じリポジトリのスキル

概要と使いどころ

Configures best-practice alerting policies for AI agents using OpenTelemetry (OTel) metrics, generating output as Terraform (.tf) configuration files. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, token usage, and quality metrics. Don't use for standard infrastructure monitoring unrelated to AI agents, or when the agent is not instrumented with OpenTelemetry (for Reliability, Cost, Safety, Security alerts). NOTE: Reliability, Cost, Safety, and Security alerts use generic OTel metrics and work across runtimes (such as Cloud Run, Vertex AI). Quality alerts rely on Vertex AI Online Monitors and are strictly bound to Vertex AI deployments.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

Deploy open models or custom weights from Model Garden to Agent Platform endpoints, check the status of an in-progress deployment operation, or clean up resources by undeploying models and deleting endpoints. Use when asked to actively deploy a model, list the Model Garden CATALOG of available models, check if a specific model is deployable (`gcloud ai model-garden models list-deployment-config`), query deployment cost, troubleshoot deployment errors (like quota limits), or undeploy/clean up endpoints. Also use when copying and deploying a 1P Tuned Model. Don't use for pure listing/discovery questions of the form "is X deployed?", "list my endpoints", or "which regions have models running?" — for those use `agent-platform-endpoint-management`. Don't use for public Vertex AI deployments (use the `vertex-deploy` skill) or for running model evaluations (use the `agent-platform-eval-flywheel` skill).

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

Manages Agent Platform serving endpoints. Use when you need to create, list, describe, update, or delete serving endpoints for model deployment on Agent Platform. Also use when troubleshooting endpoint permission, quota, or resource busy errors. Don't use for deploying models to endpoints or for running model evaluations.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

Connects to and performs inference with Google Cloud Agent Platform GenAI models, including First-Party Gemini models and Third-Party OpenMaaS models (Llama, DeepSeek, Qwen, etc.). Use when asked to perform inference, ask a model a question, run a test prompt, execute chat completions, or generate code for calling Gemini or OpenMaaS models, authenticate with GenAI SDK, OpenAI SDK, or legacy Agent Platform SDK, configure base URLs and global/regional endpoints, or troubleshoot 429 Resource Exhausted (DSQ), 400 User Validation, or 404 Not Found errors. Don't use for deploying models to endpoints or for running model evaluations.

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

Guides agents and users through migrating from Gemini API in Google AI Studio to Gemini Enterprise Agent Platform (formerly Vertex AI). Use this skill when moving applications to Google Cloud, to leverage Cloud credits, or to unify inferencing with other Cloud infrastructure (IAM, billing, telemetry).

日本語の概要は準備中です。原文の説明を表示しています。

vaila-multimodaltoolbox/vaila192026年10月8日 更新

vaila-multimodaltoolbox のスキルをすべて見る

このスキルの問題を報告する