跳到正文
cccskills
免费在 GitHub 公开

gke-ai-troubleshooting-tpu-metrics-monitoring

Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.

中文简介正在准备中。当前显示原文说明。

集成工具Kubernetes

实测评分 未实测

查看安装方法

包含的文件(3)

  • SKILL.md7.3 KB
  • references/failure_signatures.md1.4 KB
  • scripts/validate_queries.sh348 B

SKILL.md(原文)

安装前可以查看交给 AI 代理的指令内容。

GKE TPU Metrics Monitoring Guide

This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure.

Step 0: Mandatory Context

Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided {variable} placeholders:

  • {project_id}: The GCP Project ID.
  • {cluster_name}: The GKE Cluster Name.
  • {location}: The GKE Cluster Location (region or zone).
  • {node_name}: (Optional) The name of the specific GKE node.
  • {node_pool_name}: (Optional) The name of the GKE node pool.

Diagnostic Steps

Step 1: Verify TPU Runtime Metrics Configuration [Low Risk] [Auto]

Before analyzing runtime metrics, verify that the workload is configured to export them. This ensures the cluster and container environment are set up for automated metric scraping and visibility into accelerator health.

  • Action: Verify that the Pod specification and cluster meet the following prerequisites:
    • containerPort: 8431 exposed on the TPU container (required for Prometheus metric scraping).
    • JAX version 0.4.14 or later if using JAX (earlier versions do not export runtime metrics).
    • GKE version is 1.27.4-gke.900 or later (required for TPU runtime metric support).
    • GKE System Metrics are enabled on the cluster (required for Cloud Monitoring ingestion).

Step 2: Monitor TPU Runtime Metrics [Low Risk] [Auto]

If configured correctly, the following metrics are available in Cloud Monitoring (monitored resources k8s_node and k8s_container):

  • Container Metrics:
    • kubernetes.io/container/accelerator/duty_cycle: Percentage of time over the past sampling period (60 seconds) during which the TensorCores were actively processing on a TPU chip.
    • kubernetes.io/container/accelerator/memory_used: Amount of accelerator memory allocated in bytes.
    • kubernetes.io/container/accelerator/memory_total: Total accelerator memory in bytes.
  • Node Metrics:
    • kubernetes.io/node/accelerator/duty_cycle
    • kubernetes.io/node/accelerator/memory_used
    • kubernetes.io/node/accelerator/memory_total

Step 3: Check Node Status Condition [Low Risk] [Auto]

Query the status condition of GKE nodes (GKE version 1.32.1-gke.1357001 or later).

  • PromQL Query (Check if a specific node is Ready):
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"}
    
  • PromQL Query (List nodes with non-Ready conditions that are True):
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"}
    
  • PromQL Query (List nodes that are NOT Ready):
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}
    
  • PromQL Query (Fleet-wide node status):
    avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m]))
    

Step 4: Check Node Pool Status [Low Risk] [Auto]

Query the status of multi-host TPU node pools.

  • PromQL Query (Verify if a specific node pool is Running):
    kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"}
    
  • PromQL Query (Monitor node pools grouped by status):
    count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m]))
    
    Possible statuses: Provisioning, Running, Error, Reconciling, Stopping.

Step 5: Check Node Pool Availability [Low Risk] [Auto]

Query if all nodes in a multi-host TPU node pool are available.

  • PromQL Query (Check availability over time):
    avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m]))
    
    Value: 1 (True, all nodes available) or 0 (False, some nodes unavailable).

Step 6: Analyze Node Interruptions [Low Risk] [Auto]

Query the count of interruptions for GKE nodes.

  • PromQL Query (Breakdown of interruptions and causes):
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m]))
    
    Interruption Types: TerminationEvent, MaintenanceEvent, PreemptionEvent. Interruption Reasons: HostError, Eviction, AutoRepair.
  • PromQL Query (Filter for Host Maintenance events):
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m]))
    
  • PromQL Query (Interruption count aggregated by node pool):
    sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m]))
    

Step 7: Calculate Recovery and Interruption Metrics [Low Risk] [Auto]

Calculate Mean Time to Recovery (MTTR) and Mean Time Between Interruptions (MTBI) over the last 7 days.

  • PromQL Query (MTTR - Mean Time to Recovery):
    sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d]))
    
  • PromQL Query (MTBI - Mean Time Between Interruptions):
    sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d]))
    

Step 8: Monitor TPU Host Metrics [Low Risk] [Auto]

For GKE version 1.28.1-gke.1066000 or later, monitor TPU host performance.

  • Container Metrics:
    • kubernetes.io/container/accelerator/tensorcore_utilization: Current percentage of the TensorCore that is utilized.
    • kubernetes.io/container/accelerator/memory_bandwidth_utilization: Current percentage of the accelerator memory bandwidth that is being used.
  • Node Metrics:
    • kubernetes.io/node/accelerator/tensorcore_utilization
    • kubernetes.io/node/accelerator/memory_bandwidth_utilization

评价

还没有评价。欢迎分享你的使用感受。

评论

无需登录即可发表。欢迎分享使用感受、提问,或说明在哪些环境下无法运行。

还没有评论。来写第一条吧。

提交后会立即公开。请勿填写邮箱、电话等个人信息。URL 最多 2 个。

0 / 1000 字

此表单受 reCAPTCHA 保护,适用 Google 的隐私权政策和服务条款。

同一仓库的技能

Configures best-practice alerting policies for AI agents using OpenTelemetry (OTel) metrics, generating output as Terraform (.tf) configuration files. Use when analyzing, writing, or deploying alerting policies to monitor agent latency, error rates, token usage, and quality metrics. Don't use for standard infrastructure monitoring unrelated to AI agents, or when the agent is not instrumented with OpenTelemetry (for Reliability, Cost, Safety, Security alerts). NOTE: Reliability, Cost, Safety, and Security alerts use generic OTel metrics and work across runtimes (such as Cloud Run, Vertex AI). Quality alerts rely on Vertex AI Online Monitors and are strictly bound to Vertex AI deployments.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

Deploy open models or custom weights from Model Garden to Agent Platform endpoints, check the status of an in-progress deployment operation, or clean up resources by undeploying models and deleting endpoints. Use when asked to actively deploy a model, list the Model Garden CATALOG of available models, check if a specific model is deployable (`gcloud ai model-garden models list-deployment-config`), query deployment cost, troubleshoot deployment errors (like quota limits), or undeploy/clean up endpoints. Also use when copying and deploying a 1P Tuned Model. Don't use for pure listing/discovery questions of the form "is X deployed?", "list my endpoints", or "which regions have models running?" — for those use `agent-platform-endpoint-management`. Don't use for public Vertex AI deployments (use the `vertex-deploy` skill) or for running model evaluations (use the `agent-platform-eval-flywheel` skill).

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

Manages Agent Platform serving endpoints. Use when you need to create, list, describe, update, or delete serving endpoints for model deployment on Agent Platform. Also use when troubleshooting endpoint permission, quota, or resource busy errors. Don't use for deploying models to endpoints or for running model evaluations.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

Connects to and performs inference with Google Cloud Agent Platform GenAI models, including First-Party Gemini models and Third-Party OpenMaaS models (Llama, DeepSeek, Qwen, etc.). Use when asked to perform inference, ask a model a question, run a test prompt, execute chat completions, or generate code for calling Gemini or OpenMaaS models, authenticate with GenAI SDK, OpenAI SDK, or legacy Agent Platform SDK, configure base URLs and global/regional endpoints, or troubleshoot 429 Resource Exhausted (DSQ), 400 User Validation, or 404 Not Found errors. Don't use for deploying models to endpoints or for running model evaluations.

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

Guides agents and users through migrating from Gemini API in Google AI Studio to Gemini Enterprise Agent Platform (formerly Vertex AI). Use this skill when moving applications to Google Cloud, to leverage Cloud credits, or to unify inferencing with other Cloud infrastructure (IAM, billing, telemetry).

中文简介正在准备中。当前显示原文说明。

vaila-multimodaltoolbox/vaila212026年10月24日 更新

查看 vaila-multimodaltoolbox 的全部技能

报告此技能的问题