135 skills · sorted by relevance
Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.
Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.
Deploy language models with vLLM through an OpenAI-compatible API or offline batch processing. Tune response speed, throughput, and GPU memory use with quantization, which reduces numerical precision, and execution across multiple GPUs.
Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.
Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.
Generate RuVLLM settings for local language model inference and create MicroLoRA or SONA instances. Adjust their small internal weights using quality scores as feedback, with instances held in server memory for subsequent adaptation calls.
Deploy language model APIs and run batch generation with vLLM. Configure GPU memory use, quantization—lower precision model storage—and distribution across GPUs, then test response times and monitor how many requests the server handles.
Set up vLLM to serve large language models through an OpenAI-compatible API or process prompts in batches. The skill covers deployment, performance monitoring, and quantization, which reduces model memory requirements.
Plan and configure large language model training with OpenRLHF to better match human preferences. Covers training method selection, distributed GPU setup, inference acceleration with vLLM, and troubleshooting memory or training stability issues.
Apply HQQ quantization, which stores language model weights using fewer bits without calibration examples. Covers precision settings, inference backends, saving compressed models, deployment with HuggingFace Transformers or vLLM, and further training with LoRA.
Build safety checks for LLM prompts and responses using Meta's LlamaGuard. This skill covers content classification, batch moderation, serving a moderation API, and connecting checks to NeMo Guardrails.
Evaluate Hugging Face Hub models using inspect-ai or lighteval, with local inference or a provider-backed option. Choose an inference engine, start with a small test, and switch engines when model compatibility requires it.
Choose a serving container for models deployed on SageMaker. Match the model type to a compatible image, retrieve its address from the AWS catalog, and check regional availability, CPU or GPU variants, and required host settings.
Compress LLM weights using HQQ quantization, which stores weights with fewer bits without calibration datasets. Follow workflows for precision settings, model saving, inference integration, and LoRA fine-tuning of quantized models.
Helps speed up LLM text generation by comparing methods that predict and verify multiple tokens at once. Covers draft model selection, Medusa prediction heads, Lookahead Decoding settings, and deployment examples for inference services.
Run model training, fine-tuning, inference, and batch jobs on Modal’s cloud GPUs. Estimate hardware needs and cost, generate Python launch scripts, run or deploy workloads, monitor logs, and retrieve results without managing remote servers.
Diagnose failed model requests from OpenShell sandboxes to hosted APIs or local servers such as Ollama and vLLM. Check provider attachment, access rules, credentials, network layout, and client settings to identify the narrowest configuration change needed.
Guide large language model training with OpenRLHF using human preference feedback. Covers training method selection, distributed GPU setup, reward model training, and memory tuning, with examples for PPO, GRPO, and DPO.
Decision SOP for serving LLMs with vLLM. Covers PagedAttention mental model, quantization/parallelism/batching tradeoffs, OOM triage, and when NOT to use vLLM. Activates when a coder-agent is choosing or tuning an inference engine, debugging vLLM throughput/latency/OOM, or comparing vLLM against TGI/SGLang/TensorRT-LLM/llama.cpp.
The English summary is being prepared. Showing the original description.
Reduce LLM generation delays with speculative decoding, which proposes tokens ahead of verification. This skill compares small draft models, Medusa, and Lookahead Decoding, with implementation examples, tuning guidance, and deployment using vLLM.
Audit and adapt monkey patches in vllm_metax/patch/ against a target upstream revision. Decide whether to retain, update, migrate, or remove each patch; maintain patch headers and explanations; validate changes and record evidence. Use only for compatibility work on this directory, not standalone attention backend, model, kernel, or other adaptations elsewhere in vllm_metax.
The English summary is being prepared. Showing the original description.
Review and adapt vllm_metax/registry registrations, quantization configurations, CustomOps and kernel dispatch against a target vLLM revision and installed MetaX APIs. Verify registration, inherited state, weight layouts and actual backend selection. Use for registry compatibility work, not standalone attention algorithms or monkey patches.
The English summary is being prepared. Showing the original description.
Review and adapt MetaX attention backends, MLA, sparse indexers, cache layouts, and their kernel wrappers against a target vLLM revision and the actually installed MetaX component APIs. Verify dispatch, supported configurations, and GPU numerical behavior. Use for attention adaptations outside vllm_metax/patch/; do not apply monkey-patch headers or patch audit requirements.
The English summary is being prepared. Showing the original description.
Evaluate large language models on shared tests of knowledge, reasoning, and code generation with lm-evaluation-harness. Save scores and individual predictions, compare models in tables, and track performance across saved stages of model training.