'awq' 검색 결과

Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Running large language models with limited GPU memory
• Quantizing a model with domain-specific text
• Serving an AWQ model with vLLM

Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Running large models with limited GPU memory
• Compressing a model with domain-specific sample text
• Serving AWQ models with vLLM

Deploy language models with vLLM through an OpenAI-compatible API or offline batch processing. Tune response speed, throughput, and GPU memory use with quantization, which reduces numerical precision, and execution across multiple GPUs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Deploy an OpenAI-compatible language model API
• Generate text from many prompts in a batch
• Serve quantized models with limited GPU memory

Quantização de pesos consciente da ativação para compressão de LLM de 4-bit com aceleração 3x e perda mínima de acurácia. Use ao implantar modelos grandes (7B-70B) em memória GPU limitada, quando precisa de inferência mais rápida que GPTQ com melhor preservação de acurácia, ou para modelos com instruções ajustadas e multimodais. Vencedor do Prêmio Melhor Artigo MLSys 2024.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Export an approved fine-tuned model into a format suited to its hardware and runtime. Choose weight packaging and quantization, which reduces numerical precision, then compare outputs before and after export to check for regressions.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Exporting a checkpoint approved for deployment
• Choosing a model format for a target GPU or laptop
• Comparing outputs before and after export

Deploy language model APIs and run batch generation with vLLM. Configure GPU memory use, quantization—lower precision model storage—and distribution across GPUs, then test response times and monitor how many requests the server handles.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Deploying an OpenAI-compatible language model API
• Generating responses for a large set of prompts
• Fitting a large model into limited GPU memory

Think and work like an expert Machine Learning Engineer. Use when a task calls for Machine Learning Engineer judgment. Reasons from decision-policy framing, Google's Rules of ML, point-in-time data, and prefill/decode inference physics through GBDT/PyTorch baselines, vLLM/SGLang/KServe serving with FP8/AWQ quantization, hybrid-retrieval RAG, RAGAS and human-validated LLM judges, OpenTelemetry GenAI tracing, and post-Omnibus EU AI Act obligations while treating train-serve skew, temporal leakage, prompt injection, LLM nondeterminism, and degenerate feedback loops as first-class failure modes.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when working on imp's quantization formats, loaders, or dequant paths - GGUF Q4_0...Q8_0/Q*_K/IQ4/MXFP4, NVFP4 two-level scaling (Modelopt vs compressed-tensors layouts), FP8 E4M3, StorageTier, decode cache, KV-cache dtypes (auto/fp16/fp8/int8/int4/nvfp4/mxfp4), "which quant should I use", "which KV dtype", scale-factor layout, dequant kernel wiring, imp-quantize / AWQ, judging quant quality (PPL). Do NOT use for writing/optimizing GEMM/GEMV kernels (sm120-cuda-expert) or measuring quant perf (benchmark-cuda).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Serve LLMs com alta throughput usando PagedAttention do vLLM e continuous batching. Use ao fazer deploy de APIs LLM em produção, otimizar latência/throughput de inferência, ou servir modelos com memória GPU limitada. Suporta endpoints compatíveis com OpenAI, quantização (GPTQ/AWQ/FP8) e tensor parallelism.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.