Results for “fp8”

Quantize diffusion transformers with NVIDIA ModelOpt, using FP8 or NVFP4 lower-precision numbers, and prepare the saved weights for SGLang Diffusion. Follow a workflow for loading, checking output quality, and comparing speed under matching conditions.

• Preparing ModelOpt FP8 weights for SGLang Diffusion
• Checking quality after diffusion model quantization
• Comparing BF16 and quantized generation speed

Guides large language model reinforcement learning with miles, covering environment setup, low-precision training, consistency between training and inference, and speculative decoding, where a smaller model proposes tokens for a larger model to verify.

• Configure reinforcement learning for large MoE models
• Investigate instability in FP8 training
• Set up speculative decoding for faster rollout generation

Guides large language model reinforcement learning with miles, including MoE models that route work to specialized components. Covers low-precision training, matching training and inference behavior, and accelerating training response generation with draft models.

• Setting up reinforcement learning for large MoE models
• Configuring FP8 or INT4 training
• Checking consistency between training and inference

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Optimize transformer attention, the calculation that relates tokens to one another, using Flash Attention. The skill guides PyTorch integration, GPU memory reduction, performance measurement, and output comparisons for training or inference with long sequences.

• Reducing GPU memory use with long sequences
• Integrating Flash Attention into PyTorch models
• Comparing runtime and outputs before and after changes

Integrate Flash Attention into Transformer models to speed up attention, the calculation that relates input elements, while reducing GPU memory use. Covers PyTorch integration, performance measurement, output comparison, and a separate H100 optimization path.

• Optimize training with long input sequences
• Reduce memory use during long-context inference
• Measure speed and compare model outputs

Export an approved fine-tuned model into a format suited to its hardware and runtime. Choose weight packaging and quantization, which reduces numerical precision, then compare outputs before and after export to check for regressions.

• Exporting a checkpoint approved for deployment
• Choosing a model format for a target GPU or laptop
• Comparing outputs before and after export

Integrate Flash Attention into transformer models to reduce processing time and GPU memory use for long sequences. Covers PyTorch integration, advanced attention options, and benchmarks that compare speed, memory use, and output accuracy.

• Speeding up transformer training on long sequences
• Reducing GPU memory use during attention
• Adding Flash Attention to a PyTorch model

Helps configure and troubleshoot DeepSpeed for training large models across multiple devices. Covers memory optimization, parallel training, and DeepNVMe storage transfers, with examples for tuning data movement and configuring models that divide work among specialized components.

• Configuring distributed training for large models
• Tuning tensor transfers to NVMe storage
• Setting up models with specialized expert components

Adapt PyTorch training scripts for multiple GPUs or machines with Accelerate. This skill guides code changes, configuration, and launch commands, including mixed precision, which combines numerical formats, and DeepSpeed or FSDP integration for distributed training.

• Moving single-GPU training to multiple GPUs
• Configuring mixed precision training
• Setting up DeepSpeed or FSDP

Adapt existing PyTorch training scripts to run across multiple GPUs or machines with Accelerate. This skill covers device setup, distributed training configuration, and mixed precision, which uses lower-precision numbers to reduce memory use.

• Adapt a single-GPU training script for multiple GPUs
• Configure mixed precision to reduce training memory use
• Set up training with DeepSpeed or FSDP

Implement and troubleshoot large-model training with DeepSpeed across multiple GPUs. The skill covers memory optimization, parallel training, and numerical precision choices, alongside examples for storage transfers and models that divide work among specialized components.

• Review settings for distributed model training
• Tune transfers between NVMe storage and tensors
• Configure training with specialized model components

Use when working on imp's quantization formats, loaders, or dequant paths - GGUF Q4_0...Q8_0/Q*_K/IQ4/MXFP4, NVFP4 two-level scaling (Modelopt vs compressed-tensors layouts), FP8 E4M3, StorageTier, decode cache, KV-cache dtypes (auto/fp16/fp8/int8/int4/nvfp4/mxfp4), "which quant should I use", "which KV dtype", scale-factor layout, dequant kernel wiring, imp-quantize / AWQ, judging quant quality (PPL). Do NOT use for writing/optimizing GEMM/GEMV kernels (sm120-cuda-expert) or measuring quant perf (benchmark-cuda).

The English summary is being prepared. Showing the original description.

量化格式契约与位级对齐：从设备字节反推量化器的**编码公式、舍入模式、scale 粒度、退化块规则**， 并据此重实现/融合量化器（MXFP8 e8m0、int8 perblock、fp8 perchannel 等）， 用「同进程同数据 + 独立参考实现 + 中点与退化输入全覆盖」做到**逐字节精确**。 当自研量化 kernel 与框架算子**对不上**、需要逆向某个量化器的数值契约、需要判断 「量化残差算不算可接受」、或要在融合内核里复现 aclnn 量化语义时使用此 skill。 即使用户只说「量化结果和参考不一致」「这个 scale 怎么算出来的」「融合量化器精度对不上」 而未提"契约"，也应触发。**注意**：单纯选量化档位/配置（该不该开量化、开哪一档）走 dit-perf-opt；算子级性能调优与 DSL 选型走 operator-dev； 并行作用域导致的静默失效（DiT 侧「改了并行但不报错也没生效」的判定）走 dit-parallel-opt。

The English summary is being prepared. Showing the original description.

Fornece orientação para treinamento RL de nível empresarial usando miles, um fork pronto para produção do slime. Use ao treinar grandes modelos MoE com FP8/INT4, necessitando alinhamento treino-inferência ou exigindo RL especulativo para máxima taxa de transferência.

The English summary is being prepared. Showing the original description.

Otimiza atenção em transformers com Flash Attention para ganho de 2-4x em velocidade e redução de 10-20x em memória. Use ao treinar/executar transformers com sequências longas (>512 tokens), ao encontrar problemas de memória GPU com atenção, ou quando precisa de inferência mais rápida. Suporta SDPA nativo do PyTorch, biblioteca flash-attn, H100 FP8 e sliding window attention.

The English summary is being prepared. Showing the original description.

Otimiza inferência de LLM com NVIDIA TensorRT para máxima vazão e latência mínima. Use para implantação em produção em GPUs NVIDIA (A100/H100), quando você precisa de inferência 10-100x mais rápida que PyTorch, ou para servir modelos com quantização (FP8/INT4), batching em voo e escalabilidade multi-GPU.

The English summary is being prepared. Showing the original description.

Expert guidance for distributed training with DeepSpeed - ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, sparse attention

The English summary is being prepared. Showing the original description.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

The English summary is being prepared. Showing the original description.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

The English summary is being prepared. Showing the original description.

Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Use when training/running transformers with long sequences (>512 tokens), encountering GPU memory issues with attention, or need faster inference. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.

The English summary is being prepared. Showing the original description.

Set up and optimize large language model inference, or response generation, on NVIDIA GPUs. Use quantization, batch processing, and multiple GPUs to improve serving performance, and expose models through a chat completion API.

• Serving a language model through a chat completion API
• Processing many prompts in batches
• Configuring quantization to reduce memory use

Compare models available through the HuggingFace router with chat-ui configuration, research missing models, and write short English descriptions of their architecture and capabilities. Add entries to both production and development configuration files and commit the changes.

• Adding newly available router models to chat-ui
• Writing concise descriptions of model capabilities
• Updating production and development model entries together