Estimate inference memory for models on Hugging Face Hub without downloading their weights. Check GPU capacity for Safetensors or GGUF files, with optional estimates for the KV cache, memory used to retain attention data during generation.
中文简介正在准备中。当前显示英语简介。
Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub
安装前可以查看交给 AI 代理的指令内容。
hf_mem estimates the required memory for inference, including model weights and an optional KV cache, for Safetensors and GGUF for models on the Hugging Face Hub using HTTP Range requests i.e., without downloading or loading any weights locally.
uv installed (for uvx)HF_TOKEN env var or --hf-token flag (for gated or private models only)Run with --model-id pointing to the Hugging Face Hub repository which will check that it either contains Safetensors (via model.safetensors, model.safetensors.index.json if sharded, or model_index.json for Diffusers) or GGUF model weights within.
uvx hf-mem --model-id <model-id> --json-output
If the repository contains GGUF model weights in multiple precisions / quantizations, the estimations will be on a per-file basis, whereas for inference you won't load all of those but rather only a single precision. This being said, for GGUF you might as well need to provide --gguf-file to target the specific file (or path if sharded) you want to run.
uvx hf-mem --model-id <model-id> --gguf-file <file-or-path> --json-output
Additionally, hf-mem comes with an --experimental flag that will also calculate the KV cache memory requirements too, useful for large-language models, meaning it applies to LLMs (...ForCausalLM), VLMs (...ForConditionalGeneration), and GGUF models.
As per the context window, it will be read from the default or overridden with --max-model-len a la vLLM. And, same goes for the KV cache precision, which will default to the model precision unless manually set via --kv-cache-dtype a la vLLM too.
For Safetensors use as:
uvx hf-mem --model-id <model-id> --experimental [--max-model-len N] [--batch-size N] [--kv-cache-dtype auto|bfloat16|fp8|fp8_ds_mla|fp8_e4m3|fp8_e5m2|fp8_inc] --json-output
And, for GGUF use as:
uvx hf-mem --model-id <model-id> --gguf-file <file-or-path> --experimental [--max-model-len N] [--batch-size N] [--kv-cache-dtype auto|F32|F16|Q4_0|Q4_1|Q5_0|Q5_1|Q8_0|Q8_1|Q2_K|Q3_K|Q4_K|Q5_K|Q6_K|Q8_K|IQ2_XXS|IQ2_XS|IQ3_XXS|IQ1_S|IQ4_NL|IQ3_S|IQ2_S|IQ4_XS|I8|I16|I32|I64|F64|IQ1_M|BF16|TQ1_0|TQ2_0|MXFP4] --json-output
For Transformers with Safetensors weights:
uvx hf-mem --model-id MiniMaxAI/MiniMax-M2 --json-output
For Diffusers with Safetensors weights:
uvx hf-mem --model-id Qwen/Qwen-Image --json-output
For Sentence Transformers with Safetensors weights:
uvx hf-mem --model-id google/embeddinggemma-300m --json-output
With --experimental to include the KV cache estimation for LLMs and VLMs:
uvx hf-mem --model-id mistralai/Mistral-7B-v0.1 --experimental --json-output
And, for LLMs or VLMs with GGUF weights:
uvx hf-mem --model-id unsloth/Qwen3.5-397B-A17B-GGUF --gguf-file Q4_K_M --experimental --json-output
还没有评价。欢迎分享你的使用感受。
无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。
还没有评论。来写第一条吧。
Manage Hugging Face Hub through the hf command line tool. Download and upload models or datasets, organize cloud storage, manage authentication and local caches, run scheduled jobs, and deploy or maintain model inference services.
中文简介正在准备中。当前显示英语简介。
Check the AWS profile, region, account ID, and caller identity before starting AWS work. Read local configuration and validate credentials, then flag missing settings, expired access, or possible IAM restrictions associated with SSO authentication.
中文简介正在准备中。当前显示英语简介。
Prepare an isolated Python environment for SageMaker deployments and AWS automation. The skill selects a supported Python version, installs current AWS packages, and checks versions so installation and execution use the same environment.
中文简介正在准备中。当前显示英语简介。
Plan how to serve a model on Amazon SageMaker AI. The skill uses model type, traffic, response time, and cost constraints to recommend a deployment route, check instance quotas, and coordinate specialized setup and deployment skills.
中文简介正在准备中。当前显示英语简介。
Prepare SageMaker deployments and training by finding existing execution roles, checking trust settings, and reviewing required permissions. Guides role creation only when necessary and identifies permission issues that require an AWS administrator.
中文简介正在准备中。当前显示英语简介。
Deploy SageMaker endpoints, the services that answer model requests, with automatic capacity scaling, CloudWatch alarms, and resource tags. Supports real-time and asynchronous inference, scaling down to zero instances, deployment checks, and resource cleanup.
中文简介正在准备中。当前显示英语简介。