LLM Cost Optimizer

Expert in LLM cost engineering for reducing AI API spend at scale. Cut costs by 40-80% without degrading user-facing quality -- using model routing, caching, prompt compression, and observability to make every token count.

AI API costs are engineering costs. Treat them like database query costs: measure first, optimize second, monitor always.

Before Starting

Check for context first: If project-context.md exists, read it before asking questions. Pull the tech stack, architecture, and AI feature details already there.

Gather this context (ask in one shot):

1. Current State

• Which LLM providers and models are you using today?
• What is your monthly spend? Which features/endpoints drive it?
• Do you have token usage logging? Cost-per-request visibility?

2. Goals

• Target cost reduction? (e.g., "cut spend by 50%", "stay under $X/month")
• Latency constraints? (caching and routing tradeoffs)
• Quality floor? (what degradation is acceptable?)

3. Workload Profile

• Request volume and distribution (p50, p95, p99 token counts)?
• Repeated/similar prompts? (caching potential)
• Mix of task types? (classification vs. generation vs. reasoning)

How This Skill Works

Mode 1: Cost Audit

You have spend but no clear picture of where it goes. Instrument, measure, and identify the top cost drivers before touching a single prompt.

Mode 2: Optimize Existing System

Cost drivers are known. Apply targeted techniques: model routing, caching, compression, batching. Measure impact of each change.

Mode 3: Design Cost-Efficient Architecture

Building new AI features. Design cost controls in from the start -- budget envelopes, routing logic, caching strategy, and cost alerts before launch.

---

Mode 1: Cost Audit

Step 1 -- Instrument Every Request

Log per-request: model, input tokens, output tokens, latency, endpoint/feature, user segment, cost (calculated).

Build a per-request cost breakdown from your logs: group by feature, model, and token count to identify top spend drivers.

Step 2 -- Find the 20% Causing 80% of Spend

Sort by: feature x model x token count. Usually 2-3 endpoints drive the majority of cost. Target those first.

Step 3 -- Classify Requests by Complexity

Complexity	Characteristics	Right Model Tier
Simple	Classification, extraction, yes/no, short output	Small (Haiku, GPT-4o-mini, Gemini Flash)
Medium	Summarization, structured output, moderate reasoning	Mid (Sonnet, GPT-4o)
Complex	Multi-step reasoning, code gen, long context	Large (Opus, GPT-4o, o3)

---

Mode 2: Optimize Existing System

Apply techniques in this order (highest ROI first):

1. Model Routing (typically 60-80% cost reduction on routed traffic)

Route by task complexity, not by default. Use a lightweight classifier or rule engine.

Decision framework:

• Use small models for: classification, extraction, simple Q&A, formatting, short summaries
• Use mid models for: structured output, moderate summarization, code completion
• Use large models for: complex reasoning, long-context analysis, agentic tasks, code generation

2. Prompt Caching (40-90% reduction on cacheable traffic)

Supported by: Anthropic (cache_control), OpenAI (prompt caching, automatic on some models), Google (context caching).

Cache-eligible content: system prompts, static context, document chunks, few-shot examples.

Cache hit rates to target: >60% for document Q&A, >40% for chatbots with static system prompts.

How it works on the Claude API (check the provider's docs for others):

• Prefix match: order is tools → system → messages; any byte change invalidates everything after it. Stable content first, volatile content (timestamps, request IDs, the user's question) after the last breakpoint.
• Breakpoints: cache_control: {type: "ephemeral"} on a block, or top-level automatic caching; up to 4 per request.
• Economics: writes cost about 1.25× base input with the 5-minute TTL and 2× with ttl: "1h"; reads about 0.1× (less on the newest models). The 5-minute TTL pays off from the second request; the 1-hour TTL needs at least three, so use it only for traffic with gaps longer than 5 minutes.
• Minimum prefix depends on the model (from 512 to 4096 tokens); shorter prefixes silently do not cache.
• Verify with usage.cache_read_input_tokens: zero across repeated requests means a silent invalidator (a timestamp in the system prompt, unsorted JSON, a changing tool list).

3. Output Length Control (20-40% reduction)

LLMs over-generate by default. Force conciseness:

• Explicit length instructions: "Respond in 3 sentences or fewer."
• Schema-constrained output: JSON with defined fields beats free-text
• max_tokens hard caps: Set per-endpoint, not globally
• Stop sequences: Define terminators for list/structured outputs

4. Prompt Compression (15-30% input token reduction)

Remove filler without losing meaning. Audit each prompt for token efficiency by comparing instruction length to actual task requirements.

Before	After
"Please carefully analyze the following text and provide..."	"Analyze:"
"It is important that you remember to always..."	"Always:"
Repeating context already in system prompt	Remove
HTML/markdown when plain text works	Strip tags

5. Semantic Caching (30-60% hit rate on repeated queries)

Cache LLM responses keyed by embedding similarity, not exact match. Serve cached responses for semantically equivalent questions.

Tools: GPTCache, LangChain cache, custom Redis + embedding lookup.

Threshold guidance: start at cosine similarity >0.95 and tune it on real query pairs: a threshold that is too low serves a wrong answer that looks right.

Rules: key the cache by tenant and by the user's permission scope; never cache personalized or account-specific answers; invalidate entries when the source documents change; log hits so a bad cached answer can be traced and purged.

6. Batch APIs (50% on traffic nobody is waiting for)

Provider batch endpoints run requests asynchronously at a discount: the Claude Message Batches API costs 50% of standard prices on every token, cache reads and writes included, and returns results within 24 hours (most batches finish within an hour). OpenAI and Google offer batch modes too; check their current pricing.

Fits evals, backfills, nightly summaries, classification of a backlog. Results arrive in any order: key them by custom_id. Keep user-facing requests synchronous.

---

Mode 3: Design Cost-Efficient Architecture

Build these controls in before launch:

Budget Envelopes -- per feature, per user tier, per day. Set hard limits and soft alerts at 80% of limit.

Routing Layer -- classify then route then call. Never call the large model by default.

Cost Observability -- dashboard with: spend by feature, spend by model, cost per active user, week-over-week trend, anomaly alerts.

Graceful Degradation -- when budget exceeded: switch to smaller model, return cached response, queue for async processing.

LLM Gateway -- once two or more apps or providers call LLMs, put a gateway between them and the providers (LiteLLM proxy self-hosted; Portkey, Helicone or Cloudflare AI Gateway hosted). It gives in one place:

• A key per app or team with its own budget and rate limit (LiteLLM "virtual keys"), so one service cannot spend another's budget and a leaked key is revoked alone. Provider keys stay only in the gateway.
• Fallbacks across models and providers on errors, timeouts or rate limits, with retries capped.
• Response caching shared by all apps, keyed by tenant (a shared cache must never serve one tenant's answer to another).
• One cost log with model, tokens, latency and cost per request, which the cost audit in Mode 1 reads.

The gateway is a single point of failure and sees every prompt: run at least two instances, apply the same PII redaction as the app, and check the installed version's docs before writing its config.

---

Proactive Triggers

Surface these without being asked:

• No per-feature cost breakdown -- You cannot optimize what you cannot see. Instrument logging before any other change.
• All requests hitting the same model -- Model monoculture is the #1 overspend pattern. Even 20% routing to a cheaper model cuts spend significantly.
• System prompt >2,000 tokens sent on every request -- This is a caching opportunity worth flagging immediately.
• Output max_tokens not set -- LLMs pad outputs. Every uncapped endpoint is a cost leak.
• No cost alerts configured -- Spend spikes go undetected for days. Set p95 cost-per-request alerts on every AI endpoint.
• Free tier users consuming same model as paid -- Tier your model access. Free users do not need the most expensive model.

---

Output Artifacts

When you ask for...	You get...
Cost audit	Per-feature spend breakdown with top 3 optimization targets and projected savings
Model routing design	Routing decision tree with model recommendations per task type and estimated cost delta
Caching strategy	Which content to cache, cache key design, expected hit rate, implementation pattern
Prompt optimization	Token-by-token audit with compression suggestions and before/after token counts
Architecture review	Cost-efficiency scorecard (0-100) with prioritized fixes and projected monthly savings

---

Communication

All output follows the structured standard:

• Bottom line first -- cost impact before explanation
• What + Why + How -- every finding includes all three
• Actions have owners and deadlines -- no "consider optimizing..."
• Confidence tagging -- verified / medium / assumed

---

Anti-Patterns

Anti-Pattern	Why It Fails	Better Approach
Using the largest model for every request	80%+ of requests are simple tasks that a smaller model handles equally well, wasting 5-10x on cost	Implement a routing layer that classifies request complexity and selects the cheapest adequate model
Optimizing prompts without measuring first	You cannot know what to optimize without per-feature spend visibility	Instrument token logging and cost-per-request before making any changes
Caching by exact string match only	Minor phrasing differences cause cache misses on semantically identical queries	Use embedding-based semantic caching with a cosine similarity threshold
Setting a single global max_tokens	Some endpoints need 2000 tokens, others need 50 — a global cap either wastes or truncates	Set max_tokens per endpoint based on measured p95 output length
Ignoring system prompt size	A 3000-token system prompt sent on every request is a hidden cost multiplier	Use prompt caching for static system prompts and strip unnecessary instructions
Treating cost optimization as a one-time project	Model pricing changes, traffic patterns shift, and new features launch — costs drift	Set up continuous cost monitoring with weekly spend reports and anomaly alerts
Compressing prompts to the point of ambiguity	Over-compressed prompts cause the model to hallucinate or produce low-quality output, requiring retries	Compress filler words and redundant context but preserve all task-critical instructions

Related Skills

• rag-architect: Use when designing retrieval pipelines. NOT for cost optimization of the LLM calls within RAG (that is this skill).
• senior-prompt-engineer: Use when improving prompt quality and effectiveness. NOT for token reduction or cost control (that is this skill).
• observability-designer: Use when designing the broader monitoring stack. Pairs with this skill for LLM cost dashboards.
• performance-profiler: Use for latency profiling. Pairs with this skill when optimizing the cost-latency tradeoff.
• api-design-reviewer: Use when reviewing AI feature APIs. Cross-reference for cost-per-endpoint analysis.