GKE AI/ML Inference

Routing Note: For migrating existing AI workloads to GKE, open google-cloud-solution-guided-gke-ai-migration/SKILL.md. For GKE RAG with Cloud SQL or AlloyDB (pgvector), open google-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md.

This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.

MCP Tools: apply_k8s_manifest, get_k8s_resource, get_k8s_logs, get_k8s_rollout_status, describe_k8s_resource, list_k8s_events. CLI-only: gcloud container ai profiles *

When to Use

• Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
• Generate optimized Kubernetes manifests for inference
• Select GPU/TPU accelerators for model serving
• Configure autoscaling for LLM inference

Prerequisites

• A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP)
• gcloud CLI authenticated
• Sufficient GPU/TPU quota in the target region

Workflow

1. Discovery: Find Models and Hardware

# List all supported models
gcloud container ai profiles models list --quiet

# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet

# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet

2. Generate Manifest

gcloud container ai profiles manifests create \
  --model=<MODEL_NAME> \
  --model-server=<SERVER> \
  --accelerator-type=<ACCELERATOR> \
  --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml

Parameters:

• --model: Model ID (e.g., gemma-2-9b-it, llama-3-8b)
• --model-server: Inference server (vllm, tgi, triton, tensorrt-llm)
• --accelerator-type: GPU/TPU type (nvidia-l4, nvidia-tesla-a100, nvidia-h100-80gb)
• --target-ntpot-milliseconds: Target Normalized Time Per Output Token (optional, for latency optimization)

Example:

gcloud container ai profiles manifests create \
  --model=gemma-2-9b-it \
  --model-server=vllm \
  --accelerator-type=nvidia-l4 \
  --target-ntpot-milliseconds=50 --quiet > inference.yaml

3. Review and Deploy

# Review for placeholders (HF tokens, PVCs)
cat inference.yaml

# Deploy
kubectl apply -f inference.yaml

# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>

Some models require Hugging Face tokens. Create a Kubernetes Secret and reference it in the manifest.

GPU ComputeClass for Inference

For Autopilot clusters, create a ComputeClass to target GPU nodes:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: l4-inference
spec:
  priorities:
  - machineFamily: g2
    gpu:
      type: nvidia-l4
      count: 1
    minCores: 4
    minMemoryGb: 16

Accelerator Selection Guide

Accelerator	Best For	Memory	Relative Cost
NVIDIA T4	Budget inference,	16 GB	Lowest
: : lightweight legacy : : :
: : models : : :
NVIDIA L4 (G2)	Small-medium model	24 GB	Low
: : inference, video, : : :
: : graphics : : :
NVIDIA RTX PRO 6000	Multimodal AI,	96 GB	Medium
: (G4) : high-fidelity 3D, : : :
: : fine-tuning : : :
Cloud TPU v5e	Cost-effective	Varies	Medium
: : transformer inference : : :
Cloud TPU v5p	High-performance	Varies	High
: : training : : :
Cloud TPU v6e	High-efficiency next-gen	32 GB/chip	Medium-High
: (Trillium) : training & serving : : :
Cloud TPU v7x	Ultra-scale inference &	192 GB/chip	High
: (Ironwood) : agentic workflows : : :
NVIDIA A100	Large model inference,	40/80 GB	High
: : enterprise ML : : :
NVIDIA H100 / H200	Frontier model training,	80/141 GB	Highest
: : high throughput : : :
NVIDIA B200 (A4)	Blackwell-scale	192 GB	Highest
: : training, FP4 precision : : :
NVIDIA GB200 (A4X)	Rack-scale AI (Grace	Massive	Highest
: : Blackwell Superchip) : : :

Autoscaling LLM Inference

GPU-based autoscaling

Use custom metrics for GPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-server
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_duty_cycle
      target:
        type: AverageValue
        averageValue: "80"

Best practices for inference autoscaling

1. Use DCGM metrics: Golden path enables DCGM monitoring for GPU utilization metrics
2. Set appropriate minReplicas: At least 1 for always-on serving; 0 for batch/on-demand
3. Tune scale-down delay: LLM model loading is slow; use longer stabilization windows
4. Consider queue depth: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads

Optimization Tips

• Quantization: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput
• Batching: Configure model server batch size for throughput vs latency trade-off
• Tensor parallelism: Split large models across multiple GPUs within a node
• KV cache optimization: Tune --gpu-memory-utilization in vLLM for KV cache allocation

Troubleshooting

Issue	Cause	Fix
Invalid	Unsupported tuple	Re-run `gcloud container ai
: model/accelerator : : profiles list :
: combination : : --model=<MODEL>` :
GPU quota exceeded	Regional quota limit	Request quota increase or
: : : try a different region :
OOM on GPU	Model too large for	Use larger GPU, enable
: : accelerator : quantization, or use tensor :
: : : parallelism :
Slow cold start	Large model loading from	Use local SSD for model
: : registry : caching; pre-pull images :