Results for “vllm integration”

Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.

• Running large models with limited GPU memory
• Compressing a model with domain-specific sample text
• Serving AWQ models with vLLM

Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.

• Running large language models with limited GPU memory
• Quantizing a model with domain-specific text
• Serving an AWQ model with vLLM

Compress LLM weights using HQQ quantization, which stores weights with fewer bits without calibration datasets. Follow workflows for precision settings, model saving, inference integration, and LoRA fine-tuning of quantized models.

• Compressing LLMs without calibration datasets
• Serving quantized models with vLLM or HuggingFace Transformers
• Comparing bit widths and inference backends

环境安装与准备：把部署环境从零安装就绪——mindiesd 编译安装（本地昇腾直装 / SSH 推远端容器 / Docker 镜像直装）与三方推理框架全栈安装（vLLM-Omni 源码构建、DiffSynth-Engine 部署、 LightX2V editable 部署），并负责模型权重确认与下载（下载前先确认远端是否已存在）。不含特性使能与验证 （framework-integration）与 profiling（profiling-collect）；SSH 工具由 remote-access 提供。 当用户需要安装 MindIE-SD、源码构建/直装 vLLM-Omni 或 LightX2V（editable + PLATFORM=ascend_npu）、 或确认/下载模型权重时使用此技能； 即使用户只提到"把代码推到服务器""在容器里装 vllm 全栈""准备 lightx2v 调优环境"而未说昇腾， 只要上下文涉及环境安装与准备都应触发。由 model-auto-optimization S0 与 dev-workflow 部署阶段指引加载。

The English summary is being prepared. Showing the original description.

Set up LLM serving with structured responses, including JSON and outputs restricted to specified patterns. Covers reusing shared prompt calculations, tool calling, and multi-turn conversations through SGLang, with examples for server setup and application integration.

• Extract information into JSON with a defined schema
• Build agents that call tools
• Reuse shared prompts across chat requests

三方框架特性落地（原 framework-feature-enablement ＋ framework-extension-dev 合并）： 一个入口信号「框架侧特性没落地」，内部分两分支——**分支 A 框架已有 → 使能与验证** （计数契约 + 三层证据 + 使能异常回修）；**分支 B 框架缺失 → 补齐开发** （注入点 + 代码组织/注册机制 + 合入姿势：平台注册优先 / 上游 PR / fork 钉版 / monkey 备选）。 分支 A 触发词：量化/稀疏/缓存开关没生效、融合算子没命中、使能异常或精度不符、 "起 vllm 服务""模型在框架里跑不通""把 mindiesd 接到其他框架"。 分支 B 触发词：框架没 comm-stream 掩盖、框架没缓存消费者、框架不支持某 collective/mask、 feature_* NotImplemented、要给框架补一条结构性能力。 near-miss：纯安装/权重 → env-install；纯采集/分析 → profiling-collect / profiling-analyze； 纯选档（该不该开量化、开哪一档）→ dit-perf-opt； mindiesd 仓内 pattern / 算子 / 图下发开发 → dev-workflow + pattern-dev / operator-dev / aclgraph-dev。 由 model-auto-optimization 的 S1（融合接入）/S4（有损使能）/S5（训练感知）与 §0 缺口补齐策略路由， 亦由 dev-workflow 的框架接入/验证场景指引加载。

The English summary is being prepared. Showing the original description.

Quantização de pesos consciente da ativação para compressão de LLM de 4-bit com aceleração 3x e perda mínima de acurácia. Use ao implantar modelos grandes (7B-70B) em memória GPU limitada, quando precisa de inferência mais rápida que GPTQ com melhor preservação de acurácia, ou para modelos com instruções ajustadas e multimodais. Vencedor do Prêmio Melhor Artigo MLSys 2024.

The English summary is being prepared. Showing the original description.

Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.

The English summary is being prepared. Showing the original description.