'vllm' 검색 결과

Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Running large models with limited GPU memory
• Compressing a model with domain-specific sample text
• Serving AWQ models with vLLM

Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Running large language models with limited GPU memory
• Quantizing a model with domain-specific text
• Serving an AWQ model with vLLM

Deploy language models with vLLM through an OpenAI-compatible API or offline batch processing. Tune response speed, throughput, and GPU memory use with quantization, which reduces numerical precision, and execution across multiple GPUs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Deploy an OpenAI-compatible language model API
• Generate text from many prompts in a batch
• Serve quantized models with limited GPU memory

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Generate RuVLLM settings for local language model inference and create MicroLoRA or SONA instances. Adjust their small internal weights using quality scores as feedback, with instances held in server memory for subsequent adaptation calls.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Configure local language model inference
• Prepare inference tools for an air-gapped environment
• Adjust MicroLoRA using quality feedback

Deploy language model APIs and run batch generation with vLLM. Configure GPU memory use, quantization—lower precision model storage—and distribution across GPUs, then test response times and monitor how many requests the server handles.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Deploying an OpenAI-compatible language model API
• Generating responses for a large set of prompts
• Fitting a large model into limited GPU memory

Set up vLLM to serve large language models through an OpenAI-compatible API or process prompts in batches. The skill covers deployment, performance monitoring, and quantization, which reduces model memory requirements.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Deploying an OpenAI-compatible language model API
• Generating text from large batches of prompts
• Serving large models with limited GPU memory

Plan and configure large language model training with OpenRLHF to better match human preferences. Covers training method selection, distributed GPU setup, inference acceleration with vLLM, and troubleshooting memory or training stability issues.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Choosing a method for training models on human preferences
• Setting up PPO training with a reward model
• Configuring GRPO training without a critic model

Build safety checks for LLM prompts and responses using Meta's LlamaGuard. This skill covers content classification, batch moderation, serving a moderation API, and connecting checks to NeMo Guardrails.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Screening user prompts before sending them to an LLM
• Checking generated responses before displaying them
• Building an API for content safety checks

Apply HQQ quantization, which stores language model weights using fewer bits without calibration examples. Covers precision settings, inference backends, saving compressed models, deployment with HuggingFace Transformers or vLLM, and further training with LoRA.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compressing a model without calibration data
• Comparing 4-bit and 2-bit quantization settings
• Running quantized models with HuggingFace Transformers or vLLM

Evaluate Hugging Face Hub models using inspect-ai or lighteval, with local inference or a provider-backed option. Choose an inference engine, start with a small test, and switch engines when model compatibility requires it.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Run benchmark evaluations on a local GPU
• Choose an evaluation framework and inference engine
• Check a small sample before a full evaluation

Choose a serving container for models deployed on SageMaker. Match the model type to a compatible image, retrieve its address from the AWS catalog, and check regional availability, CPU or GPU variants, and required host settings.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Choose a container for an LLM deployment
• Select images for embeddings and rerankers
• Check regional image addresses and GPU compatibility

Compress LLM weights using HQQ quantization, which stores weights with fewer bits without calibration datasets. Follow workflows for precision settings, model saving, inference integration, and LoRA fine-tuning of quantized models.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compressing LLMs without calibration datasets
• Serving quantized models with vLLM or HuggingFace Transformers
• Comparing bit widths and inference backends

Helps speed up LLM text generation by comparing methods that predict and verify multiple tokens at once. Covers draft model selection, Medusa prediction heads, Lookahead Decoding settings, and deployment examples for inference services.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Reducing chatbot response latency
• Comparing methods for faster code generation
• Selecting a small draft model for a larger LLM

Run model training, fine-tuning, inference, and batch jobs on Modal’s cloud GPUs. Estimate hardware needs and cost, generate Python launch scripts, run or deploy workloads, monitor logs, and retrieve results without managing remote servers.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Test CUDA code without a local GPU
• Run model training or LoRA fine-tuning on cloud GPUs
• Deploy a model inference API or vLLM service

Diagnose failed model requests from OpenShell sandboxes to hosted APIs or local servers such as Ollama and vLLM. Check provider attachment, access rules, credentials, network layout, and client settings to identify the narrowest configuration change needed.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Diagnose failed requests to hosted model APIs
• Troubleshoot connections to Ollama or LM Studio
• Investigate credential and endpoint mismatch errors

Guide large language model training with OpenRLHF using human preference feedback. Covers training method selection, distributed GPU setup, reward model training, and memory tuning, with examples for PPO, GRPO, and DPO.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Choosing a training method for human preference alignment
• Setting up distributed training across multiple GPUs
• Training a reward model before PPO

Decision SOP for serving LLMs with vLLM. Covers PagedAttention mental model, quantization/parallelism/batching tradeoffs, OOM triage, and when NOT to use vLLM. Activates when a coder-agent is choosing or tuning an inference engine, debugging vLLM throughput/latency/OOM, or comparing vLLM against TGI/SGLang/TensorRT-LLM/llama.cpp.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Reduce LLM generation delays with speculative decoding, which proposes tokens ahead of verification. This skill compares small draft models, Medusa, and Lookahead Decoding, with implementation examples, tuning guidance, and deployment using vLLM.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Reducing response delays in chatbots
• Speeding up code generation
• Choosing a small model to draft candidate tokens

Review and adapt MetaX attention backends, MLA, sparse indexers, cache layouts, and their kernel wrappers against a target vLLM revision and the actually installed MetaX component APIs. Verify dispatch, supported configurations, and GPU numerical behavior. Use for attention adaptations outside vllm_metax/patch/; do not apply monkey-patch headers or patch audit requirements.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Audit and adapt monkey patches in vllm_metax/patch/ against a target upstream revision. Decide whether to retain, update, migrate, or remove each patch; maintain patch headers and explanations; validate changes and record evidence. Use only for compatibility work on this directory, not standalone attention backend, model, kernel, or other adaptations elsewhere in vllm_metax.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Review and adapt vllm_metax/registry registrations, quantization configurations, CustomOps and kernel dispatch against a target vLLM revision and installed MetaX APIs. Verify registration, inherited state, weight layouts and actual backend selection. Use for registry compatibility work, not standalone attention algorithms or monkey patches.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate large language models on shared tests of knowledge, reasoning, and code generation with lm-evaluation-harness. Save scores and individual predictions, compare models in tables, and track performance across saved stages of model training.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Evaluating models on MMLU and GSM8K
• Building comparison tables for multiple models
• Tracking scores across training checkpoints