Results for “fp16”

Triage incorrect MNN outputs and regressions by matching symptoms to focused debugging guides. Use backend comparisons, minimal reproductions, and controlled experiments to investigate memory issues, quantization, concurrency, numeric limits, caches, and inconsistent results across runs.

• Investigating garbled or degraded MNN output
• Comparing results across execution backends
• Tracking regressions after optimization changes

Helps configure and troubleshoot DeepSpeed for training large models across multiple devices. Covers memory optimization, parallel training, and DeepNVMe storage transfers, with examples for tuning data movement and configuring models that divide work among specialized components.

• Configuring distributed training for large models
• Tuning tensor transfers to NVMe storage
• Setting up models with specialized expert components

Diagnose errors and incorrect inference results in MNN's QNN backend for Qualcomm HTP processors. Compare intermediate calculation data against CPU results to locate the first faulty operator, distinguish bugs from precision limits, and guide operator support and performance fixes.

• Investigate QNN runtime and model conversion errors
• Find the first operator producing incorrect results
• Distinguish implementation bugs from numerical precision limits

Implement and troubleshoot large-model training with DeepSpeed across multiple GPUs. The skill covers memory optimization, parallel training, and numerical precision choices, alongside examples for storage transfers and models that divide work among specialized components.

• Review settings for distributed model training
• Tune transfers between NVMe storage and tensors
• Configure training with specialized model components

Expert guidance for distributed training with DeepSpeed - ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, sparse attention

The English summary is being prepared. Showing the original description.

Develop lightweight CUDA operations for SGLang using JIT, which compiles code on first use. A tensor-scaling example explains implementation conventions, input validation, memory access helpers, and when to choose this approach.

• Add a lightweight CUDA operation to SGLang
• Choose between JIT and ahead-of-time compilation
• Validate tensor inputs in a kernel launcher

Add a CUDA/C++ GPU operation to sgl-kernel using AOT, meaning compilation before use. Follow the implementation through PyTorch registration, Python API exposure, SGLang integration, correctness tests, and benchmarks that account for cache reuse.

• Adding a CUDA/C++ operation to sgl-kernel
• Registering an operation with PyTorch and exposing a Python API
• Testing numerical results and invalid inputs

Build safety checks for LLM prompts and responses using Meta's LlamaGuard moderation model. Classify potentially harmful content, block flagged messages, serve checks through an API, and integrate them with NeMo Guardrails.

• Checking prompts before sending them to an LLM
• Filtering responses before displaying them
• Building a moderation API

Reduce the memory needed to run large language models with GPTQ quantization, which stores model weights at lower precision. Covers loading compressed models, quantizing your own models, choosing execution backends, and connecting to memory-efficient fine-tuning.

• Run a language model within limited GPU memory
• Load a pre-quantized model for text generation
• Quantize and save your own trained model

Adapt PyTorch training scripts for multiple GPUs or machines with Accelerate. This skill guides code changes, configuration, and launch commands, including mixed precision, which combines numerical formats, and DeepSpeed or FSDP integration for distributed training.

• Moving single-GPU training to multiple GPUs
• Configuring mixed precision training
• Setting up DeepSpeed or FSDP

Adapt existing PyTorch training scripts to run across multiple GPUs or machines with Accelerate. This skill covers device setup, distributed training configuration, and mixed precision, which uses lower-precision numbers to reduce memory use.

• Adapt a single-GPU training script for multiple GPUs
• Configure mixed precision to reduce training memory use
• Set up training with DeepSpeed or FSDP

Compress large language models to 4-bit weights to reduce GPU memory needs and accelerate text generation. Covers loading compressed models, converting your own models, choosing settings, and preparing memory-efficient fine-tuning with transformers and PEFT.

• Run language models with limited GPU memory
• Convert your own model to 4-bit weights
• Compare compression settings for quality and speed

Guides PyTorch model training and fine-tuning, from choosing architectures to configuring optimizers and learning rates. Provides troubleshooting steps for unstable loss and memory shortages, plus methods for improving GPU throughput and comparing experiments.

• Choose model architectures for the available data type and scale
• Configure optimizers and learning-rate schedules
• Investigate loss spikes, NaNs, and GPU memory shortages

Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.

The English summary is being prepared. Showing the original description.

Use when working on imp's quantization formats, loaders, or dequant paths - GGUF Q4_0...Q8_0/Q*_K/IQ4/MXFP4, NVFP4 two-level scaling (Modelopt vs compressed-tensors layouts), FP8 E4M3, StorageTier, decode cache, KV-cache dtypes (auto/fp16/fp8/int8/int4/nvfp4/mxfp4), "which quant should I use", "which KV dtype", scale-factor layout, dequant kernel wiring, imp-quantize / AWQ, judging quant quality (PPL). Do NOT use for writing/optimizing GEMM/GEMV kernels (sm120-cuda-expert) or measuring quant perf (benchmark-cuda).

The English summary is being prepared. Showing the original description.

Use when writing, reviewing, or optimizing CUDA kernels targeting sm_120a (RTX 5090 / Consumer Blackwell, GB202) in the imp inference engine. Triggers on CUDA/PTX kernel code, shared-memory layout, bank conflicts, tensor-core MMA (mxf4nvf4, tf32, 3xTF32/3xFP16), GEMV/GEMM/attention/quantization/GDN-scan kernels, decode tok/s under expected, kernel emitting HMMA instead of mxf4nvf4, occupancy, register pressure, __launch_bounds__, spills, PDL. Pair with `benchmark-cuda` for measurement and `check-degeneration` after hot-path changes.

The English summary is being prepared. Showing the original description.

Quantização pós-treinamento em 4-bits para LLMs com perda mínima de precisão. Use para implantar modelos grandes (70B, 405B) em GPUs de consumo, quando você precisa de redução de memória 4× com <2% de degradação de perplexidade, ou para inferência mais rápida (aceleração de 3-4×) vs FP16. Integra com transformers e PEFT para fine-tuning QLoRA.

The English summary is being prepared. Showing the original description.

API de treinamento distribuído mais simples. 4 linhas para adicionar suporte distribuído a qualquer script PyTorch. API unificada para DeepSpeed/FSDP/Megatron/DDP. Posicionamento automático de device, precisão mista (FP16/BF16/FP8). Config interativo, comando de launch único. Padrão do ecossistema HuggingFace.

The English summary is being prepared. Showing the original description.

Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.

The English summary is being prepared. Showing the original description.

Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.

The English summary is being prepared. Showing the original description.

Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.

The English summary is being prepared. Showing the original description.

Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.

The English summary is being prepared. Showing the original description.