Set up and optimize large language model inference, or response generation, on NVIDIA GPUs. Use quantization, batch processing, and multiple GPUs to improve serving performance, and expose models through a chat completion API.
中文简介正在准备中。当前显示英语简介。
High-throughput LLM inference on NVIDIA GPUs.
安装前可以查看交给 AI 代理的指令内容。
NVIDIA's open-source library for optimizing LLM inference with high performance on NVIDIA GPUs.
Use TensorRT-LLM when:
Use vLLM instead when:
Use llama.cpp instead when:
# Docker (recommended) — images are on NGC (nvcr.io), not Docker Hub.
# Replace x.y.z with the desired version (e.g. 1.2.1). Browse tags on NGC:
# https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags
docker pull nvcr.io/nvidia/tensorrt-llm/release:x.y.z
# pip install (current stable GA)
pip install tensorrt_llm
# Requires CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.12
from tensorrt_llm import LLM, SamplingParams
# Initialize model
llm = LLM(model="meta-llama/Meta-Llama-3-8B")
# Configure sampling
sampling_params = SamplingParams(
max_tokens=100,
temperature=0.7,
top_p=0.9
)
# Generate
prompts = ["Explain quantum computing"]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(output.text)
# Start server (automatic model download and compilation)
# Tensor parallelism across 4 GPUs
trtllm-serve meta-llama/Meta-Llama-3-8B \
--tp_size 4 \
--max_batch_size 256 \
--max_num_tokens 4096
# Client request
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-8B",
"messages": [{"role": "user", "content": "Hello!"}],
"temperature": 0.7,
"max_tokens": 100
}'
from tensorrt_llm import LLM
# Load FP8 quantized model (2× faster, 50% memory)
llm = LLM(
model="meta-llama/Meta-Llama-3-70B",
dtype="fp8",
max_num_tokens=8192
)
# Inference same as before
outputs = llm.generate(["Summarize this article..."])
# Tensor parallelism across 8 GPUs
llm = LLM(
model="meta-llama/Meta-Llama-3-405B",
tensor_parallel_size=8,
dtype="fp8"
)
# Process 100 prompts efficiently
prompts = [f"Question {i}: ..." for i in range(100)]
outputs = llm.generate(
prompts,
sampling_params=SamplingParams(max_tokens=200)
)
# Automatic in-flight batching for maximum throughput
Meta Llama 3-8B (H100 GPU):
Llama 3-70B (8× A100 80GB):
还没有评价。欢迎分享你的使用感受。
无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。
还没有评论。来写第一条吧。
Set up the 1Password CLI and use stored passwords or API keys in commands and configuration templates. Choose an authentication method, retrieve secret references, and pass credentials to processes through environment variables.
中文简介正在准备中。当前显示英语简介。
Build an Excel workbook linking the income statement, balance sheet, and cash flow statement. Populate historical figures, create formula-based forecasts, compare scenarios, and check that balances and cash totals agree across all three statements.
中文简介正在准备中。当前显示英语简介。
Adapt existing PyTorch training scripts to use multiple GPUs or machines with limited code changes. The skill guides configuration, launch commands, mixed precision, which uses lower-precision numbers, and DeepSpeed or FSDP settings for distributing training work and memory.
中文简介正在准备中。当前显示英语简介。
Configure Hermes to run AI models through Actual Computer on your own hardware. Set up either the hosted relay or a local connection, select models, verify responses, and troubleshoot request errors or empty output.
中文简介正在准备中。当前显示英语简介。
Test an app from the perspective of a frustrated, less technical user. Find confusing steps and barriers, separate actionable usability problems from personal complaints, and produce screenshots, suggested fixes, and prioritized tickets.
中文简介正在准备中。当前显示英语简介。
Resolve Git merge conflicts between two agents by comparing both changes and their stated goals. Combine compatible work, explain choices where designs differ, and verify the merged result with project builds or tests.
中文简介正在准备中。当前显示英语简介。