Results for “int4”

Guides large language model reinforcement learning with miles, including MoE models that route work to specialized components. Covers low-precision training, matching training and inference behavior, and accelerating training response generation with draft models.

• Setting up reinforcement learning for large MoE models
• Configuring FP8 or INT4 training
• Checking consistency between training and inference

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Guides large language model reinforcement learning with miles, covering environment setup, low-precision training, consistency between training and inference, and speculative decoding, where a smaller model proposes tokens for a larger model to verify.

• Configure reinforcement learning for large MoE models
• Investigate instability in FP8 training
• Set up speculative decoding for faster rollout generation

Export an approved fine-tuned model into a format suited to its hardware and runtime. Choose weight packaging and quantization, which reduces numerical precision, then compare outputs before and after export to check for regressions.

• Exporting a checkpoint approved for deployment
• Choosing a model format for a target GPU or laptop
• Comparing outputs before and after export

Fornece orientação para treinamento RL de nível empresarial usando miles, um fork pronto para produção do slime. Use ao treinar grandes modelos MoE com FP8/INT4, necessitando alinhamento treino-inferência ou exigindo RL especulativo para máxima taxa de transferência.

The English summary is being prepared. Showing the original description.

Otimiza inferência de LLM com NVIDIA TensorRT para máxima vazão e latência mínima. Use para implantação em produção em GPUs NVIDIA (A100/H100), quando você precisa de inferência 10-100x mais rápida que PyTorch, ou para servir modelos com quantização (FP8/INT4), batching em voo e escalabilidade multi-GPU.

The English summary is being prepared. Showing the original description.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

The English summary is being prepared. Showing the original description.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

The English summary is being prepared. Showing the original description.

Use when working on imp's quantization formats, loaders, or dequant paths - GGUF Q4_0...Q8_0/Q*_K/IQ4/MXFP4, NVFP4 two-level scaling (Modelopt vs compressed-tensors layouts), FP8 E4M3, StorageTier, decode cache, KV-cache dtypes (auto/fp16/fp8/int8/int4/nvfp4/mxfp4), "which quant should I use", "which KV dtype", scale-factor layout, dequant kernel wiring, imp-quantize / AWQ, judging quant quality (PPL). Do NOT use for writing/optimizing GEMM/GEMV kernels (sm120-cuda-expert) or measuring quant perf (benchmark-cuda).

The English summary is being prepared. Showing the original description.

Deploy machine learning models to edge devices using Google AI Edge Gallery, TensorFlow Lite, ONNX Runtime, and MediaPipe. Covers model quantization (INT8/INT4), on-device inference with Gemma 4 models, Android/iOS deployment via AI Edge Gallery, hardware delegate selection (GPU/NPU/DSP), and performance benchmarking on constrained devices. Use when deploying models to mobile phones, IoT devices, or embedded systems where cloud inference is impractical due to latency, cost, or connectivity constraints.

The English summary is being prepared. Showing the original description.

Iterate on a GenAI notebook against the self-hosted stack via the genai-stack CLI (config dirs, auth, subdomains, quantization, GPU/VRAM). Arguments: <notebook|service> [--service comfyui|forge|vllm] [--quant int4|fp8] [--validate] [--bg]

The English summary is being prepared. Showing the original description.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

The English summary is being prepared. Showing the original description.