Results for “gpu”

Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.

• Running large language models with limited GPU memory
• Quantizing a model with domain-specific text
• Serving an AWQ model with vLLM

Use this skill for hands-on DOCA GPI programming — wiring a GPU-Packet-Initiator context so a CUDA kernel drives RDMA queues directly from GPU memory without host CPU mediation. Covers picking GPI vs doca-gpunetio, the doca_gpi / domain / channel object model, the GPU-side handle handoff (doca_gpu_gpi_channel*), attaching GPU memory to a GPI domain, the domain and channel attribute objects, and debugging DOCA_ERROR_* from doca_gpi_* calls. Trigger even when the user does not explicitly mention "DOCA GPI" — implicit phrasings include "my CUDA kernel needs to post RDMA directly from GPU memory", "DOCA_ERROR_* from doca_gpi_gpu_channel_get", "how do I hand a GPU handle to my CUDA kernel", "how many channels can a GPI domain hold", or "GPU kernel driving RDMA without the host CPU on the path". Refuse and route elsewhere for the doca-gpunetio Send/Receive surface, the doca-rdma queue lifecycle, DPA-side initiation (doca-rdmi), or the CUDA programming model — those belong to other skills.

The English summary is being prepared. Showing the original description.

AI demos and GPU compute with Gradio Spaces and Hugging Face Spaces ZeroGPU. Use when writing or reviewing code that uses `@spaces.GPU`, configuring `python_version` or `requirements.txt` for a ZeroGPU Space, or handling ZeroGPU-specific code constraints — pickle-based process isolation, `gr.State` semantics across the worker boundary, no `torch.compile` (use AoTI instead), CUDA wheel-only builds (no `nvcc` at build or runtime), large vs xlarge sizing, and dynamic duration callables. Make sure to use this skill whenever the user mentions ZeroGPU, `@spaces.GPU`, or the `spaces` Python package, or hits ZeroGPU-specific code errors like `PicklingError` across the worker boundary, `illegal duration`, or `flash-attn` wheel-build failures — even when the user does not explicitly ask for ZeroGPU coding guidance. Trigger on `import spaces` or `@spaces.GPU` in code.

The English summary is being prepared. Showing the original description.

Investigate recurring SGLang automated test failures by comparing logs, commit history, and GPU environments. Narrow down code changes or machine-specific problems, optionally reproduce failures remotely, and summarize the evidence in a structured report.

• Finding the change behind a recurring test failure
• Checking whether failures depend on a GPU or runner
• Reproducing a test failure on a remote GPU server

Build similarity search with FAISS to find nearby matches in large collections of numeric vectors. The skill guides index selection, memory reduction, GPU use, and saving indexes while balancing search speed and accuracy.

• Find similar items in large vector datasets
• Choose an index for search speed and accuracy
• Reduce memory use for vector search

Plan and configure large language model training with OpenRLHF to better match human preferences. Covers training method selection, distributed GPU setup, inference acceleration with vLLM, and troubleshooting memory or training stability issues.

• Choosing a method for training models on human preferences
• Setting up PPO training with a reward model
• Configuring GRPO training without a critic model

Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.

• Pretraining Llama 3.1 8B on eight GPUs
• Configuring multi-node training with SLURM
• Enabling Float8 training on H100 GPUs

Adapt PyTorch training scripts for multiple GPUs or machines with Accelerate. This skill guides code changes, configuration, and launch commands, including mixed precision, which combines numerical formats, and DeepSpeed or FSDP integration for distributed training.

• Moving single-GPU training to multiple GPUs
• Configuring mixed precision training
• Setting up DeepSpeed or FSDP

Investigate interruptions affecting GPU and TPU workloads on GKE using maintenance labels, monitoring queries, logs, and node scheduling restrictions. The skill helps determine whether host maintenance caused a disruption and recommends workload protection measures when maintenance is confirmed or expected.

• Investigating unexpected GPU or TPU workload interruptions
• Checking upcoming host maintenance on GKE nodes
• Tracking maintenance interruptions with PromQL

Improve MNN's Vulkan backend by separating CPU scheduling bottlenecks from GPU computation costs. It guides operator and shader changes, new feature integration with compatibility fallbacks, and device testing to verify both correctness and end-to-end performance.

• Finding CPU or GPU bottlenecks in Vulkan inference
• Optimizing a specific operator or shader
• Integrating cooperative matrix or subgroup features

Build similarity search with FAISS to find nearby matches in large collections of numeric vectors. The skill guides index selection, memory reduction, GPU use, and saving indexes while balancing search speed and accuracy.

• Find similar items in large vector datasets
• Choose an index for search speed and accuracy
• Reduce memory use for vector search

Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.

• Pretraining Llama 3.1 8B on eight GPUs
• Configuring multi-node training with SLURM
• Enabling Float8 training on H100 GPUs

Guides you through running large language models on CPUs, Apple Silicon, and AMD or Intel GPUs. Covers choosing GGUF model files, reducing memory through quantization, and starting an OpenAI-compatible chat server.

• Running a local LLM on an Apple Silicon Mac
• Trying LLM inference on a CPU-only machine
• Choosing a model format that fits available memory

Adapt existing PyTorch training scripts to run across multiple GPUs or machines with Accelerate. This skill covers device setup, distributed training configuration, and mixed precision, which uses lower-precision numbers to reduce memory use.

• Adapt a single-GPU training script for multiple GPUs
• Configure mixed precision to reduce training memory use
• Set up training with DeepSpeed or FSDP

Build and maintain machine learning apps on Hugging Face Spaces. Choose a suitable app framework and computing hardware, load models, deploy files, and check logs and outputs to diagnose build or runtime problems.

• Publishing a machine learning demo
• Adapting an existing Gradio app for ZeroGPU
• Diagnosing Space build or runtime failures

Plan how to split AI models and workloads across multiple GPUs. Compare memory needs, computation, communication, and waiting time to estimate completion times and choose a layout suited to response time, throughput, or cost goals.

• Running models that exceed a single GPU's memory
• Comparing ways to split work across GPUs
• Estimating communication costs within and between servers

Set up vLLM to serve large language models through an OpenAI-compatible API or process prompts in batches. The skill covers deployment, performance monitoring, and quantization, which reduces model memory requirements.

• Deploying an OpenAI-compatible language model API
• Generating text from large batches of prompts
• Serving large models with limited GPU memory

Generate complete songs locally from lyrics and tags describing mood, instruments, or genre. The skill guides setup, input formatting, memory-saving settings, and MP3 output using the open-source HeartMuLa music models.

• Turn original lyrics into a complete song
• Try different moods and instruments with tags
• Generate music locally or offline

Investigate SGLang runs that freeze across multiple GPUs. It uses stack traces, communication logs, GPU memory dumps, and comparisons between processes to locate state differences, trace their causes, and guide fixes.

• Locate where a multi-GPU inference run freezes
• Compare logs from distributed processes
• Trace the first difference in process state

Analyze LLM serving traces to identify GPU time usage, opportunities to run operations concurrently, and known patterns for combining operations. Produce three consistent tables from saved traces or live captures on supported servers.

• Identify GPU operations that dominate inference time
• Compare input processing and token generation
• Trace GPU operations back to Python code

Configure machine learning training and batch jobs across multiple clouds with SkyPilot. The skill guides GPU selection, lower-cost cloud and region choices, and automatic recovery when discounted spot instances are interrupted.

• Choose a lower-cost cloud for GPU training
• Run long training jobs with recovery after interruptions
• Configure distributed training across multiple machines

Organize PyTorch training code with LightningModule and Trainer. Follow examples for validation, logging, model saving, and distributed training across multiple GPUs, with callbacks that add monitoring or stop training when progress stalls.

• Converting an existing PyTorch training loop
• Adding validation, logs, and model checkpoints
• Configuring training across multiple GPUs

Run batches of machine learning experiments on remote GPUs over SSH, varying settings and random seeds. The skill schedules jobs, tracks progress, retries GPU memory failures, and cleans up leftover sessions before starting subsequent batches.

• Run experiments across multiple random seeds
• Test combinations of model settings
• Chain teacher training and student distillation

Configure machine learning training and batch jobs across multiple clouds with SkyPilot. The skill guides GPU selection, lower-cost cloud and region choices, and automatic recovery when discounted spot instances are interrupted.

• Choose a lower-cost cloud for GPU training
• Run long training jobs with recovery after interruptions
• Configure distributed training across multiple machines