106 skills · sorted by relevance
Convert LLMs to GGUF and reduce model size through quantization, which stores weights at lower numerical precision. Prepare models for CPU or Apple Silicon inference, compare compression levels, and integrate them with local AI tools.
Apply HQQ quantization, which stores language model weights using fewer bits without calibration examples. Covers precision settings, inference backends, saving compressed models, deployment with HuggingFace Transformers or vLLM, and further training with LoRA.
Compress LLM weights using HQQ quantization, which stores weights with fewer bits without calibration datasets. Follow workflows for precision settings, model saving, inference integration, and LoRA fine-tuning of quantized models.
Convert language models to GGUF and reduce their size through quantization, which stores model weights at lower precision. Guides cover compression choices, local inference on CPUs and Apple Silicon, and integration with tools such as Ollama.
Reduce the memory needed to run large language models with GPTQ quantization, which stores model weights at lower precision. Covers loading compressed models, quantizing your own models, choosing execution backends, and connecting to memory-efficient fine-tuning.
Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.
Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.
Helps reduce GPU memory use by loading large language models in 4-bit or 8-bit formats, a technique called quantization. Covers bitsandbytes configuration, memory checks, QLoRA fine-tuning, and training with memory-saving optimizers.
Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4/3/2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.
The English summary is being prepared. Showing the original description.
Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.
Deploy language models with vLLM through an OpenAI-compatible API or offline batch processing. Tune response speed, throughput, and GPU memory use with quantization, which reduces numerical precision, and execution across multiple GPUs.
Run language models locally with llama.cpp on CPUs, Apple Silicon, or AMD and Intel GPUs. The skill guides model setup, memory-saving GGUF format choices, interactive chat, and serving models through an OpenAI-compatible API.
Deploy language model APIs and run batch generation with vLLM. Configure GPU memory use, quantization—lower precision model storage—and distribution across GPUs, then test response times and monitor how many requests the server handles.
Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.
Guides you through running large language models on CPUs, Apple Silicon, and AMD or Intel GPUs. Covers choosing GGUF model files, reducing memory through quantization, and starting an OpenAI-compatible chat server.
Compress large language models to 4-bit weights to reduce GPU memory needs and accelerate text generation. Covers loading compressed models, converting your own models, choosing settings, and preparing memory-efficient fine-tuning with transformers and PEFT.
End-to-end automated model quantization and accuracy tuning workflow. Use when user asks for automated model quantization and accuracy tuning, e.g. "自动量化", "量化调优", "一键量化", "精度调优", etc. 当用户给出多卡（≥2 卡）卡号时，量化阶段通过 EP 并行适配（msmodelslim-ep-parallel-adaptation）保证多卡 EP 并行，调优主流程仍在本 Skill。
The English summary is being prepared. Showing the original description.
Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.
The English summary is being prepared. Showing the original description.
Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4/3/2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.
The English summary is being prepared. Showing the original description.
GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.
The English summary is being prepared. Showing the original description.
GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.
The English summary is being prepared. Showing the original description.
Set up vLLM to serve large language models through an OpenAI-compatible API or process prompts in batches. The skill covers deployment, performance monitoring, and quantization, which reduces model memory requirements.
Set up and optimize large language model inference, or response generation, on NVIDIA GPUs. Use quantization, batch processing, and multiple GPUs to improve serving performance, and expose models through a chat completion API.
Tune AgentDB memory use and search speed through quantization, which stores vectors more compactly, search indexes, caching, and batch operations. Includes configuration examples and benchmarks to help balance accuracy, speed, and storage as datasets grow.