“fast inference”的搜索结果

Reduce LLM weight usage through pruning, which removes selected model parameters. Explore Wanda and SparseGPT for compression without retraining, choose hardware-friendly patterns for faster inference, and compare the compressed model's accuracy with the original.

中文简介正在准备中。当前显示英语简介。

• Compressing an LLM without retraining
• Comparing pruning methods
• Exploring faster inference on supported hardware

Build evolutionary trees from related nucleotide or protein sequences using MAFFT with IQ-TREE or FastTree. Review branch support, a measure of evidence for each grouping, and produce tree summaries and optional visualizations.

中文简介正在准备中。当前显示英语简介。

• Build gene trees from related nucleotide sequences
• Compare evolutionary relationships within protein families
• Review branch support and analysis logs

Set up LLM serving with structured responses, including JSON and outputs restricted to specified patterns. Covers reusing shared prompt calculations, tool calling, and multi-turn conversations through SGLang, with examples for server setup and application integration.

中文简介正在准备中。当前显示英语简介。

• Extract information into JSON with a defined schema
• Build agents that call tools
• Reuse shared prompts across chat requests

Helps speed up LLM text generation by comparing methods that predict and verify multiple tokens at once. Covers draft model selection, Medusa prediction heads, Lookahead Decoding settings, and deployment examples for inference services.

中文简介正在准备中。当前显示英语简介。

• Reducing chatbot response latency
• Comparing methods for faster code generation
• Selecting a small draft model for a larger LLM

Build LLM serving workflows that produce JSON or other required formats. The skill covers reusing computation for shared prompts, tool calls, and ongoing conversations, with examples for server setup and structured output.

中文简介正在准备中。当前显示英语简介。

• Extracting article details into JSON
• Building agents that call tools
• Reusing shared prompts across conversations

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

中文简介正在准备中。当前显示英语简介。

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

中文简介正在准备中。当前显示英语简介。

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Guides LLM compression through pruning, which removes selected model weights. Covers Wanda and SparseGPT methods without retraining, accuracy comparisons, and structured weight patterns that can accelerate inference on compatible hardware.

中文简介正在准备中。当前显示英语简介。

• Trying model compression without retraining
• Comparing accuracy before and after pruning
• Choosing between Wanda and SparseGPT

Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.

中文简介正在准备中。当前显示原文说明。

Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.

中文简介正在准备中。当前显示英语简介。

• Running large models with limited GPU memory
• Compressing a model with domain-specific sample text
• Serving AWQ models with vLLM

Reduce LLM generation delays with speculative decoding, which proposes tokens ahead of verification. This skill compares small draft models, Medusa, and Lookahead Decoding, with implementation examples, tuning guidance, and deployment using vLLM.

中文简介正在准备中。当前显示英语简介。

• Reducing response delays in chatbots
• Speeding up code generation
• Choosing a small model to draft candidate tokens

Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.

中文简介正在准备中。当前显示英语简介。

• Running large language models with limited GPU memory
• Quantizing a model with domain-specific text
• Serving an AWQ model with vLLM

Reduce LLM size and accelerate inference using pruning techniques like Wanda and SparseGPT. Use when compressing models without retraining, achieving 50% sparsity with minimal accuracy loss, or enabling faster inference on hardware accelerators. Covers unstructured pruning, structured pruning, N:M sparsity, magnitude pruning, and one-shot methods.

中文简介正在准备中。当前显示原文说明。

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

中文简介正在准备中。当前显示原文说明。

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

中文简介正在准备中。当前显示原文说明。

Guides large language model reinforcement learning with miles, covering environment setup, low-precision training, consistency between training and inference, and speculative decoding, where a smaller model proposes tokens for a larger model to verify.

中文简介正在准备中。当前显示英语简介。

• Configure reinforcement learning for large MoE models
• Investigate instability in FP8 training
• Set up speculative decoding for faster rollout generation

Compress LLM weights using HQQ quantization, which stores weights with fewer bits without calibration datasets. Follow workflows for precision settings, model saving, inference integration, and LoRA fine-tuning of quantized models.

中文简介正在准备中。当前显示英语简介。

• Compressing LLMs without calibration datasets
• Serving quantized models with vLLM or HuggingFace Transformers
• Comparing bit widths and inference backends

Reduce the memory needed to run large language models with GPTQ quantization, which stores model weights at lower precision. Covers loading compressed models, quantizing your own models, choosing execution backends, and connecting to memory-efficient fine-tuning.

中文简介正在准备中。当前显示英语简介。

• Run a language model within limited GPU memory
• Load a pre-quantized model for text generation
• Quantize and save your own trained model

Build visual AI systems that detect objects or divide images into labeled regions. This skill guides dataset preparation, model selection, training, evaluation, inference optimization, and deployment using frameworks such as PyTorch and Ultralytics.

中文简介正在准备中。当前显示英语简介。

• Build an object detection pipeline
• Clean images and convert annotation formats
• Optimize a trained model for deployment

Guides adapting OpenPI robot models to custom datasets and serving them to produce actions from observations. Covers JAX and PyTorch training, checkpoint conversion, and troubleshooting for normalization statistics and GPU memory.

中文简介正在准备中。当前显示英语简介。

• Fine-tune OpenPI models on custom robot datasets
• Convert JAX checkpoints to PyTorch
• Serve policies for ALOHA, DROID, or LIBERO

Compress large language models to 4-bit weights to reduce GPU memory needs and accelerate text generation. Covers loading compressed models, converting your own models, choosing settings, and preparing memory-efficient fine-tuning with transformers and PEFT.

中文简介正在准备中。当前显示英语简介。

• Run language models with limited GPU memory
• Convert your own model to 4-bit weights
• Compare compression settings for quality and speed

Cerebras Inference is a hosted LLM API that runs open models on Cerebras wafer-scale chips at thousands of tokens per second, through an OpenAI-compatible endpoint. Use when a user wants the lowest-latency chat, code completion or agent loop, asks how to call the Cerebras API or SDK, needs streaming, tool calling, strict JSON schema output or reasoning control, or hits a "model not found" error from a retired Cerebras model ID.

中文简介正在准备中。当前显示原文说明。

Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.

中文简介正在准备中。当前显示原文说明。

Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.

中文简介正在准备中。当前显示原文说明。