“vllm”的搜索结果

Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.

中文简介正在准备中。当前显示英语简介。

• Running large models with limited GPU memory
• Compressing a model with domain-specific sample text
• Serving AWQ models with vLLM

Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.

中文简介正在准备中。当前显示英语简介。

• Running large language models with limited GPU memory
• Quantizing a model with domain-specific text
• Serving an AWQ model with vLLM

协助配置vLLM，将大语言模型作为API提供服务或批量生成文本，涵盖部署、性能监控、量化和多GPU分工，也用于排查显存不足与响应缓慢。

• 提供兼容OpenAI的API
• 批量处理大量提示词
• 在有限显存下运行模型

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

中文简介正在准备中。当前显示英语简介。

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Deploy language models with vLLM through an OpenAI-compatible API or offline batch processing. Tune response speed, throughput, and GPU memory use with quantization, which reduces numerical precision, and execution across multiple GPUs.

中文简介正在准备中。当前显示英语简介。

• Deploy an OpenAI-compatible language model API
• Generate text from many prompts in a batch
• Serve quantized models with limited GPU memory

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

中文简介正在准备中。当前显示英语简介。

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Generate RuVLLM settings for local language model inference and create MicroLoRA or SONA instances. Adjust their small internal weights using quality scores as feedback, with instances held in server memory for subsequent adaptation calls.

中文简介正在准备中。当前显示英语简介。

• Configure local language model inference
• Prepare inference tools for an air-gapped environment
• Adjust MicroLoRA using quality feedback

Deploy language model APIs and run batch generation with vLLM. Configure GPU memory use, quantization—lower precision model storage—and distribution across GPUs, then test response times and monitor how many requests the server handles.

中文简介正在准备中。当前显示英语简介。

• Deploying an OpenAI-compatible language model API
• Generating responses for a large set of prompts
• Fitting a large model into limited GPU memory

Plan and configure large language model training with OpenRLHF to better match human preferences. Covers training method selection, distributed GPU setup, inference acceleration with vLLM, and troubleshooting memory or training stability issues.

中文简介正在准备中。当前显示英语简介。

• Choosing a method for training models on human preferences
• Setting up PPO training with a reward model
• Configuring GRPO training without a critic model

Build safety checks for LLM prompts and responses using Meta's LlamaGuard. This skill covers content classification, batch moderation, serving a moderation API, and connecting checks to NeMo Guardrails.

中文简介正在准备中。当前显示英语简介。

• Screening user prompts before sending them to an LLM
• Checking generated responses before displaying them
• Building an API for content safety checks

Apply HQQ quantization, which stores language model weights using fewer bits without calibration examples. Covers precision settings, inference backends, saving compressed models, deployment with HuggingFace Transformers or vLLM, and further training with LoRA.

中文简介正在准备中。当前显示英语简介。

• Compressing a model without calibration data
• Comparing 4-bit and 2-bit quantization settings
• Running quantized models with HuggingFace Transformers or vLLM

Evaluate Hugging Face Hub models using inspect-ai or lighteval, with local inference or a provider-backed option. Choose an inference engine, start with a small test, and switch engines when model compatibility requires it.

中文简介正在准备中。当前显示英语简介。

• Run benchmark evaluations on a local GPU
• Choose an evaluation framework and inference engine
• Check a small sample before a full evaluation

Choose a serving container for models deployed on SageMaker. Match the model type to a compatible image, retrieve its address from the AWS catalog, and check regional availability, CPU or GPU variants, and required host settings.

中文简介正在准备中。当前显示英语简介。

• Choose a container for an LLM deployment
• Select images for embeddings and rerankers
• Check regional image addresses and GPU compatibility

Compress LLM weights using HQQ quantization, which stores weights with fewer bits without calibration datasets. Follow workflows for precision settings, model saving, inference integration, and LoRA fine-tuning of quantized models.

中文简介正在准备中。当前显示英语简介。

• Compressing LLMs without calibration datasets
• Serving quantized models with vLLM or HuggingFace Transformers
• Comparing bit widths and inference backends

Helps speed up LLM text generation by comparing methods that predict and verify multiple tokens at once. Covers draft model selection, Medusa prediction heads, Lookahead Decoding settings, and deployment examples for inference services.

中文简介正在准备中。当前显示英语简介。

• Reducing chatbot response latency
• Comparing methods for faster code generation
• Selecting a small draft model for a larger LLM

Run model training, fine-tuning, inference, and batch jobs on Modal’s cloud GPUs. Estimate hardware needs and cost, generate Python launch scripts, run or deploy workloads, monitor logs, and retrieve results without managing remote servers.

中文简介正在准备中。当前显示英语简介。

• Test CUDA code without a local GPU
• Run model training or LoRA fine-tuning on cloud GPUs
• Deploy a model inference API or vLLM service

Diagnose failed model requests from OpenShell sandboxes to hosted APIs or local servers such as Ollama and vLLM. Check provider attachment, access rules, credentials, network layout, and client settings to identify the narrowest configuration change needed.

中文简介正在准备中。当前显示英语简介。

• Diagnose failed requests to hosted model APIs
• Troubleshoot connections to Ollama or LM Studio
• Investigate credential and endpoint mismatch errors

Guide large language model training with OpenRLHF using human preference feedback. Covers training method selection, distributed GPU setup, reward model training, and memory tuning, with examples for PPO, GRPO, and DPO.

中文简介正在准备中。当前显示英语简介。

• Choosing a training method for human preference alignment
• Setting up distributed training across multiple GPUs
• Training a reward model before PPO

Decision SOP for serving LLMs with vLLM. Covers PagedAttention mental model, quantization/parallelism/batching tradeoffs, OOM triage, and when NOT to use vLLM. Activates when a coder-agent is choosing or tuning an inference engine, debugging vLLM throughput/latency/OOM, or comparing vLLM against TGI/SGLang/TensorRT-LLM/llama.cpp.

中文简介正在准备中。当前显示原文说明。

Reduce LLM generation delays with speculative decoding, which proposes tokens ahead of verification. This skill compares small draft models, Medusa, and Lookahead Decoding, with implementation examples, tuning guidance, and deployment using vLLM.

中文简介正在准备中。当前显示英语简介。

• Reducing response delays in chatbots
• Speeding up code generation
• Choosing a small model to draft candidate tokens

Review and adapt vllm_metax/registry registrations, quantization configurations, CustomOps and kernel dispatch against a target vLLM revision and installed MetaX APIs. Verify registration, inherited state, weight layouts and actual backend selection. Use for registry compatibility work, not standalone attention algorithms or monkey patches.

中文简介正在准备中。当前显示原文说明。

Review and adapt MetaX attention backends, MLA, sparse indexers, cache layouts, and their kernel wrappers against a target vLLM revision and the actually installed MetaX component APIs. Verify dispatch, supported configurations, and GPU numerical behavior. Use for attention adaptations outside vllm_metax/patch/; do not apply monkey-patch headers or patch audit requirements.

中文简介正在准备中。当前显示原文说明。

Audit and adapt monkey patches in vllm_metax/patch/ against a target upstream revision. Decide whether to retain, update, migrate, or remove each patch; maintain patch headers and explanations; validate changes and record evidence. Use only for compatibility work on this directory, not standalone attention backend, model, kernel, or other adaptations elsewhere in vllm_metax.

中文简介正在准备中。当前显示原文说明。

使用统一任务评估大语言模型的知识、推理和代码生成能力，生成多模型对比表及训练阶段的成绩趋势，并保存评估结果和每道题的回答。

• 使用MMLU和GSM8K评估模型
• 生成多个模型的成绩对比表
• 跟踪训练期间的模型成绩