Inference Serving

This is a Hermes-native inference-serving workflow skill.

Why This Exists

inference-serving exists so serving an LLM runs as one decided, gated, measured process instead of scattered flag folklore: the engine choice is a table, the deployment is an idempotent runbook whose only completion evidence is the observed verification, and the benchmark speaks the standard metric vocabulary.

Do Not Use When

• A new model generation needs recognition, calibration, routing, and pricing onboarding; use model-optimization.
• The user wants their own machine's model routing or providers configured; use model-setup.
• The question is whether a coding runtime/executor can run at all; use executor-runtime-readiness.
• The goal is application or system performance rather than the serving endpoint itself; use ultraperf.

Examples

Good example:

• Prompt: Serve Qwen on our two A100s for the team and tell me if prefix caching is worth turning on.
• Expected behavior: Engine verdict (vLLM, TP as a power of two), quantization check, the k8s or docker runbook with its gates and verification, then the prefix-cache A/B protocol with hit-rate assumptions recorded - numbers only from observed runs.
• Why: Serving plus a measured tuning question is exactly the decide-deploy-measure process this workflow owns.

Bad example:

• Prompt: Just tell me the endpoint is fast enough, we already know it works.
• Expected behavior: Refuse the unmeasured claim; run the benchmark protocol against the stated SLO or report the capacity question as unanswered.
• Why: A fast-enough claim without a load shape and observed results is the folklore this skill replaces.

Completion Checklist

• The engine/quantization verdict names the situation-table row it came from and the rejected options.
• Every runbook step's status is prepared or observed, never assumed, and the port invariant was honored.
• Benchmark numbers carry metrics, load shape, dataset, SLO, and saved metadata, or are not reported.
• Anything the workflow started for measurement was stopped, and credentials never appear in artifacts.

Recovery Notes

• If the hardware truth is unknown, probe it first (GPU inventory, VRAM) instead of assuming the engine.
• If deployment verification fails, walk the failure ladder (toolkit, shared memory, permissions, token) before editing manifests.
• If a benchmark misses the verify targets, go to the symptom->flag table and re-measure one change at a time.

Workflow Lane

• Current lane: Research and company ops (product-docs, source-finder, web-research, research, model-optimization, inference-serving, model-finetuning, research-brief, +20 more) - research, signals, ops, and briefings.
• If intent belongs to another lane, hand back to oh-my-hermes or name the adjacent workflow.
• Shared product, routing, compatibility, and evidence rules: omh-routing/references/skill-common-rail.md.

Use When

Use when a model needs to be served - engine and quantization chosen, docker or Kubernetes deployment prepared as a gated runbook, or the endpoint measured with the TTFT/TPOT/ITL/goodput protocol - and the user wants the process, not an ad-hoc command guess.

Strong routing signals: `inference-serving`, `inference serving`, `serve this model`, `serve the model`, `model serving`, `serving endpoint`, `vllm`, `llama.cpp`, `llama cpp`, `serve with vllm`, `deploy vllm`, `vllm deployment`, `serving benchmark`, `benchmark the endpoint`, `prefix caching benchmark`, `gguf quantization`, `which quantization`, `모델 서빙`, `모델 서빙해줘`, `모델 배포해서 서빙`, `서빙 벤치마크`, `vllm 배포`, `vllm 서빙`, `추론 서버 띄워줘`, `모델 띄워줘`

Catalog Metadata

Category: operations Phase: inference-serving Hermes role: operator Quality tier: observed-command-gated Reasoning demand: light

Quality bar:

• Decide before deploying: engine from the situation table (vLLM for multi-user NVIDIA APIs, llama.cpp for CPU/Apple Silicon/edge, TensorRT-LLM only with ops budget), quantization to match (AWQ/GPTQ/FP8 vs the GGUF ladder with Q4_K_M default), tensor parallel a power of two.
• Deploy as the gated runbook: docker's three load-bearing flags (--ipc=host, HF cache mount, HF_TOKEN) or the Kubernetes five-step (secret gate, existing-deployment gate, apply, rollout+readiness verify, summary+smoke); the port invariant touches four places or it did not change the port.
• Troubleshoot from the symptom table first - slow TTFT to prefix caching/chunked prefill, OOM to gpu-memory-utilization/max-model-len/quantization - before inventing flags.
• Measure with the protocol: TTFT/TPOT/ITL/E2EL as mean/median/P99, goodput against an explicit SLO, one load shape per run, results saved with metadata; the full contract is omh-inference-serving/references/serving-bench.md.
• Report observed-only: each runbook step is prepared until its command's exit status and output are seen.

Handoff policy:

Keep engine/quantization decisions, runbook preparation, and benchmark design in Hermes; the commands run through the operator's terminal with observed evidence, and repository changes (deploy manifests, benchmark harnesses) are coding work for the selected executor lane. A runbook or benchmark plan is prepared_not_observed until its commands' results are seen.

Required inputs:

• the model id(s) and where the weights live (HF id, local path, gated or not)
• the hardware truth: GPUs and VRAM, or CPU/Apple Silicon, and single- vs multi-user load
• the delivery surface: docker, Kubernetes, or bare process, and the port/ingress constraints
• for benchmarks: the SLO (TTFT/TPOT bounds) and the load shape the number must represent

Expected outputs:

• engine and quantization verdict from the decision tables, with the rejected options named
• deployment runbook with its gates (secret, existing-deployment), verification commands, and the four-places port invariant
• benchmark plan naming metrics, load shape, dataset, and metadata to save
• observed-only status: what ran, what was verified, what stays prepared

Artifact expectations:

• serving decision and runbook per omh-inference-serving/references/serving-runbooks.md
• benchmark protocol per omh-inference-serving/references/serving-bench.md
• result files with metadata only after observed runs

Safety rules:

• Never claim the server is up without the observed rollout/readiness or smoke-request evidence.
• Never write credentials into runbooks or results; tokens are referenced (HF_TOKEN, a named secret), never inlined.
• A healthy probe is not a benchmark; a benchmark number without its load shape and metadata is not reported.
• If the workflow started a server for a benchmark, the workflow stops it.

Runtime Evidence

Record observed delegation results; otherwise return not_available or not_observed. Prepared OMH routing is not execution, review, CI, merge-readiness, or merge evidence.

• Treat wrapper memory/context summaries as advisory local context, not proof of opaque Hermes memory reads or changes. Preserve workflow intent and stop conditions; verify before claiming completion. Reply in the user's own words and the host's own voice: its SOUL.md persona owns reply language, tone, speech level, and sentence endings, progress updates included (where it sets no language, use the one the user wrote in), and OMH shapes structure and content only; OMH's record terms (surface, lane, wrapper, handoff, evidence boundary, not_observed) stay in records and tool calls, never in the sentence the user reads unless they ask about one; and when a stop condition or a decision the user owns ends the turn, offer the next action as a question rather than declaring what will not be done.

Use Hermes-native subagent/delegation features when available: native subagents -> Hermes delegation when available, otherwise sequential lanes.

Shared product, compatibility, topology, memory, harness, and execution rules: omh-routing/references/skill-common-rail.md. Load it when applicable; otherwise name an unavailable capability.