“inference serving”的搜索结果

针对在Itô已预订GPU上进行模型服务的咨询，说明当前尚未支持的范围，并整理未来所需的配置、资格验证及确认事项；目前不会启动模型服务。

• 确认预订Itô GPU后的服务支持情况
• 了解OpenAI兼容端点的限制
• 查看未来模型服务所需配置

[omh] Self-hosted LLM serving on GPUs: choose the serving engine and quantization from decision tables, prepare deployment as an idempotent runbook with observed-only verification, and measure the endpoint with the standard TTFT/TPOT/goodput protocol. Use when the user says: inference-serving, inference serving, serve this model, serve the model, model serving, serving endpoint, vllm, llama.cpp.

中文简介正在准备中。当前显示原文说明。

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

中文简介正在准备中。当前显示英语简介。

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Configure LLM inference and serving on NVIDIA GPUs with TensorRT-LLM. Covers quantization, which reduces numerical precision, batching requests during generation, and distributing models across GPUs to improve processing capacity and response times.

中文简介正在准备中。当前显示英语简介。

• Setting up an LLM server on NVIDIA GPUs
• Configuring FP8 or INT4 quantization
• Distributing large models across multiple GPUs

Set up LLM serving with structured responses, including JSON and outputs restricted to specified patterns. Covers reusing shared prompt calculations, tool calling, and multi-turn conversations through SGLang, with examples for server setup and application integration.

中文简介正在准备中。当前显示英语简介。

• Extract information into JSON with a defined schema
• Build agents that call tools
• Reuse shared prompts across chat requests

Build LLM serving workflows that produce JSON or other required formats. The skill covers reusing computation for shared prompts, tool calls, and ongoing conversations, with examples for server setup and structured output.

中文简介正在准备中。当前显示英语简介。

• Extracting article details into JSON
• Building agents that call tools
• Reusing shared prompts across conversations

Plan how to serve a model on Amazon SageMaker AI. The skill uses model type, traffic, response time, and cost constraints to recommend a deployment route, check instance quotas, and coordinate specialized setup and deployment skills.

中文简介正在准备中。当前显示英语简介。

• Plan AWS hosting for a language model
• Choose a serving route for an embedding model
• Plan asynchronous inference for image generation

Deploy AI models on GKE with suitable GPUs or TPUs and model servers. Generate deployment configuration, adjust automatic scaling, and tune inference—the process of producing model responses—for memory use, throughput, and response time.

中文简介正在准备中。当前显示英语简介。

• Deploying Gemma or Llama on GKE
• Choosing GPUs or TPUs for model serving
• Configuring automatic scaling for LLM requests

Deploy machine learning models to production serving infrastructure using MLflow, BentoML, or Seldon Core with REST/gRPC endpoints, implement autoscaling, monitoring, and A/B testing capabilities for high-performance model inference at scale. Use when deploying trained models for real-time inference, setting up REST or gRPC prediction APIs, implementing autoscaling for variable load, running A/B tests between model versions, or migrating from batch to real-time inference.

中文简介正在准备中。当前显示原文说明。

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

中文简介正在准备中。当前显示原文说明。

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

中文简介正在准备中。当前显示原文说明。

Build Python workloads on Modal for cloud execution, including GPU model training and inference, web APIs, scheduled jobs, and parallel batch processing. Configure container environments, persistent storage, and automatic scaling through the Modal SDK.

中文简介正在准备中。当前显示英语简介。

• Serving an AI model through a web API
• Running model training or fine-tuning on cloud GPUs
• Processing independent batch inputs in parallel

Deploy language models with vLLM through an OpenAI-compatible API or offline batch processing. Tune response speed, throughput, and GPU memory use with quantization, which reduces numerical precision, and execution across multiple GPUs.

中文简介正在准备中。当前显示英语简介。

• Deploy an OpenAI-compatible language model API
• Generate text from many prompts in a batch
• Serve quantized models with limited GPU memory

Run language models locally with llama.cpp on CPUs, Apple Silicon, or AMD and Intel GPUs. The skill guides model setup, memory-saving GGUF format choices, interactive chat, and serving models through an OpenAI-compatible API.

中文简介正在准备中。当前显示英语简介。

• Running a language model on a CPU-only computer
• Setting up local chat on an Apple Silicon Mac
• Choosing model quantization to fit available memory

Deploy language model APIs and run batch generation with vLLM. Configure GPU memory use, quantization—lower precision model storage—and distribution across GPUs, then test response times and monitor how many requests the server handles.

中文简介正在准备中。当前显示英语简介。

• Deploying an OpenAI-compatible language model API
• Generating responses for a large set of prompts
• Fitting a large model into limited GPU memory

Guides adapting OpenPI robot models to custom datasets and serving them to produce actions from observations. Covers JAX and PyTorch training, checkpoint conversion, and troubleshooting for normalization statistics and GPU memory.

中文简介正在准备中。当前显示英语简介。

• Fine-tune OpenPI models on custom robot datasets
• Convert JAX checkpoints to PyTorch
• Serve policies for ALOHA, DROID, or LIBERO

Guides you through running large language models on CPUs, Apple Silicon, and AMD or Intel GPUs. Covers choosing GGUF model files, reducing memory through quantization, and starting an OpenAI-compatible chat server.

中文简介正在准备中。当前显示英语简介。

• Running a local LLM on an Apple Silicon Mac
• Trying LLM inference on a CPU-only machine
• Choosing a model format that fits available memory

Compress large language model weights to four bits with AWQ, a method that uses model activation patterns to limit quality loss. Follow workflows for quantization, loading compressed models, and serving them with vLLM using less GPU memory.

中文简介正在准备中。当前显示英语简介。

• Running large language models with limited GPU memory
• Quantizing a model with domain-specific text
• Serving an AWQ model with vLLM

LLM and ML model deployment for inference. Use when serving models in production, building AI APIs, or optimizing inference. Covers vLLM (LLM serving), TensorRT-LLM (GPU optimization), Ollama (local), BentoML (ML deployment), Triton (multi-model), LangChain (orchestration), LlamaIndex (RAG), and streaming patterns.

中文简介正在准备中。当前显示原文说明。

Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.

中文简介正在准备中。当前显示原文说明。

帮助在NVIDIA GPU上加速大语言模型的响应生成，并通过API提供服务。支持量化、批量处理和多GPU分布运行，以调整显存占用与服务性能。

• 通过聊天API提供语言模型服务
• 批量处理大量提示词
• 通过量化减少显存占用

Compress LLM weights using HQQ quantization, which stores weights with fewer bits without calibration datasets. Follow workflows for precision settings, model saving, inference integration, and LoRA fine-tuning of quantized models.

中文简介正在准备中。当前显示英语简介。

• Compressing LLMs without calibration datasets
• Serving quantized models with vLLM or HuggingFace Transformers
• Comparing bit widths and inference backends

Compress large language models to 4-bit weights to reduce GPU memory use and accelerate inference. Covers loading compressed models, configuring quantization, and using Transformers or vLLM, with guidance on hardware requirements and backend choices.

中文简介正在准备中。当前显示英语简介。

• Running large models with limited GPU memory
• Compressing a model with domain-specific sample text
• Serving AWQ models with vLLM

Choose a serving container for models deployed on SageMaker. Match the model type to a compatible image, retrieve its address from the AWS catalog, and check regional availability, CPU or GPU variants, and required host settings.

中文简介正在准备中。当前显示英语简介。

• Choose a container for an LLM deployment
• Select images for embeddings and rerankers
• Check regional image addresses and GPU compatibility