AI-Augmented Product Engineering

Quick Reference

Concern	Defaults
LLM API integration	Vercel AI SDK by default; use a direct provider SDK when its capabilities or smaller abstraction fit better (see Integration and Gateway Choice)
Streaming responses	SSE / ReadableStream from the SDK's server streaming call; streaming mode matched to guardrail risk (see Streaming Architecture)
Structured output	Schema-constrained output or tool calling, one schema definition (e.g. Zod) validated in code; contract rules in ai-prompt-engineering
Chat interface	The SDK's chat UI hook, or a custom streaming UI
AI-assisted forms	Inline suggestions, auto-complete, content generation
Guardrails	Input/output filtering, content moderation, PII detection
Cost management	Token counting, caching (semantic + exact), model routing
Multi-provider	Thin internal router for one product; consider a Portkey-class gateway for shared routing or policy (see Integration and Gateway Choice)
Evaluation	Human feedback, LLM-as-judge, A/B testing AI variants
RAG in products	Vector search + context injection (see also ai-rag for deeper patterns)

When NOT to Use This Skill

• LLM lifecycle management (fine-tuning, deployment, monitoring) → ai-llm
• Agent system architecture and orchestration → ai-agents
• Prompt engineering techniques and patterns → ai-prompt-engineering
• RAG system architecture (indexing, retrieval, chunking) → ai-rag
• ML model training and data science → ai-ml-data-science
• MLOps and model serving infrastructure → ai-mlops
• Building MCP servers and tool protocols → agents-mcp

When NOT to Add AI At All

Not every "AI feature" request should become one. Apply this filter before the decision tree below:

• Deterministic logic solves it. If the mapping from input to output is a rule, a regex, a lookup table, or a small classifier that can be trained offline, an LLM call adds latency, cost, and non-determinism for no accuracy gain. Reach for an LLM when the input space is open-ended natural language or the task requires judgment a rule set cannot encode.
• The eval bar cannot be met. If nobody can articulate what "correct" looks like well enough to write 20-50 test cases with expected outputs, the team cannot tell if the feature works, regresses, or is safe to ship. Build the eval set before writing the first prompt (see ai-evals) — "we'll know it when we see it" is not a launch gate.
• The failure mode is unacceptable and unrecoverable. High-stakes one-shot actions (irreversible financial transfers, medical dosing, legal filings) need a human-in-the-loop confirmation step even when the model is highly accurate; if the product cannot afford any rate of wrong output and cannot insert a checkpoint, do not automate that step with an LLM.
• A cheaper, boring solution already ships the value. Autocomplete from historical data, templated responses, or a search index often satisfy the underlying user need without a model call. Prototype the non-AI version first if it is cheap to build — it is also the fallback path required by the guardrail rules below.

Integration and Gateway Choice

Need	Default	Change when
Product chat, streaming, extraction	Vercel AI SDK; direct provider SDK when its capabilities or a smaller abstraction matter	Use an embedded copilot shell when its UI is central to the feature; use agent orchestration only for stateful multi-step work
Moderation and prompt-injection checks	Provider moderation if available, otherwise a tested classifier	Add a dedicated guard service if adversarial testing shows the simpler gate misses required cases
Evaluation and traces	Keep product-specific criteria in code; use existing observability or an LLM tracing service	Build storage only when retention, access, or query requirements cannot be met
Provider routing	Thin internal router for one product	Consider a Portkey-class gateway when several teams need common budgets or policy; first check vendor ownership/control continuity and assign owners for keys, prompt/log retention, residency, export, and rollback

Keep prompts, schemas, eval criteria, and business rules under product ownership even when infrastructure is bought.

Workflow

1. Classify the feature shape: chat, generation, extraction, search, or agent-adjacent UX.
2. Confirm the product boundary and route architecture-heavy or retrieval-heavy work to the adjacent skill when needed.
3. Pick the primary pattern from the decision tree, then define the request contract, latency target, and safety checks.
4. Apply the relevant implementation guidance for streaming, structure, cost, and guardrails.
5. Verify current provider capabilities with the navigation references and fact-checking rules before final recommendations.

Decision Tree

What kind of AI feature?
├─ Chat / conversational UI
│  ├─ Web app → app AI SDK (chat UI hook + server streaming call)
│  │  ├─ Conversation history → Store in DB, not just client state
│  │  ├─ Streaming markdown → Progressive render with remark/rehype
│  │  └─ Multi-turn with tools → Tool results in message history
│  └─ Mobile / native → Direct SSE consumption + custom UI
├─ Content generation ("write for me")
│  ├─ Short-form (titles, descriptions) → Single generation + schema
│  └─ Long-form (articles, reports)
│     └─ Draft → Review → Edit → Apply pattern with undo support
├─ Inline suggestions (autocomplete)
│  ├─ Latency-critical → Smallest, fastest model tier that passes the evals
│  ├─ UI pattern → Ghost text, accept with Tab, dismiss with Esc
│  └─ Trigger → Debounce input (300-500ms), cancel in-flight requests
├─ Data extraction / classification
│  ├─ Structured output with schema validation in code
│  ├─ Batch processing → Queue + worker pattern
│  └─ Confidence scores → Include in schema, filter by threshold
├─ Search / Q&A over content
│  └─ RAG pattern (see ai-rag) + this skill for product integration
└─ Agent features (multi-step autonomous)
   └─ ai-agents for architecture, this skill for product UX integration

Streaming Architecture

Stream server-side (the SDK's streaming call, piped as SSE/ReadableStream) and render progressively client-side, but match the streaming mode to guardrail risk:

• Low-risk surfaces (drafting, brainstorming, internal tools): stream tokens and run output checks in parallel on sentence or chunk boundaries. On a violation, stop the stream, retract the rendered text in the UI, and log it.
• High-stakes or regulated output (medical, legal, financial, PII-bearing, anything a user may act on): buffer server-side until the output guardrail passes, then release. Show a progress state to cover the latency.
• Fail closed applies to the release, not to the tokens. A guardrail cannot block tokens already rendered, so a surface that must never show unchecked text is a buffered surface.

Handle mid-stream errors inline with a context-preserving retry, make "stop generating" cancel the upstream call and any pending tool executions, and respect backpressure. Details: references/product-integration-patterns.md.

Structured Output Wiring

• Bind the schema with the SDK's schema-output facility and validate the parsed object. Look up the current function name in the SDK's migration guide or changelog when writing code; these APIs are renamed between major versions. One schema definition (e.g. Zod) gives both types and runtime validation.
• Check the provider's strict-schema subset before deployment. Compile the canonical schema for each provider/model path and reject unsupported keywords or shapes; never silently weaken required constraints. Provider schema acceptance is a deployment gate, while application validation still gates persistence and side effects.
• Stream partial objects for display only. Validate the complete object before persistence or any side effect.
• Handle non-success responses explicitly. A refusal or token-limit truncation can bypass the expected object shape. Inspect the provider/SDK completion status before parsing; record a typed refusal or incomplete result, then retry only when safe and bounded or route to review.
• Contract design lives in ai-prompt-engineering: schema field order, closed enums with unknown, strict schema mode vs tool calling, discriminated unions, and the one-retry repair loop are in ai-prompt-engineering core-patterns. Schema-constrained modes guarantee syntax where supported, not correctness.

Conversation & Context Management

Store message history server-side, version system prompts in code, and keep tool results in history.

Default history policy (switch to a pure sliding window only for stateless chit-chat):

1. Fixed budget for the system prompt plus tool definitions; never truncate them.
2. Then pinned user facts (stated preferences, constraints, entity IDs).
3. Then the last N turns verbatim.
4. When history exceeds about half of the remaining budget, summarize the older turns into one block.
5. Keep the full transcript server-side, so a summary can be redone or audited.

Memory design and how to verify that compaction kept what matters: ai-context-layer context-hygiene.

Regenerate and retry must not re-execute side effects. Give every side-effecting tool call an idempotency key derived from (conversation, turn, call index), and make the tool honor it. Regenerate and stream-retry replay the stored tool results instead of calling write tools again; "stop generating" cancels pending tool executions, not only the token stream. Details: references/product-integration-patterns.md.

Cost Control

Control	Default rule
Token estimation	Estimate input tokens before sending; warn or truncate at budget threshold; use tiktoken or provider tokenizer
Caching	Exact-match cache for deterministic queries; semantic cache (embeddings) for FAQ-style; set TTLs
Model routing	Small, fast tier for classification/extraction/autocomplete; frontier tier for complex reasoning/long-form; confirm each choice with evals
Usage tracking	Track tokens per user, per feature, per model; budget alerts at 70% and 90% of monthly allocation
Rate limiting	Apply per user tier at the application layer; return clear error messages with upgrade paths
Prompt optimization	Audit system prompts for verbosity; measure quality vs. length trade-offs

See references/rollout-and-observability.md for cost dashboards, eval loops, and feature-flag rollout patterns.

Prompt-caching break-even rule

Caching pays only once a cached prefix is reused enough to amortize the cache-write premium. Cost model (per prefix token, prices as multiples of the normal input price, which is 1): the first call writes the cache at multiplier w (> 1); each later call that hits the cache before it expires reads at multiplier r (< 1); without caching every call pays 1.

• Derivation: for 1 write + k reads, no-cache cost = (1 + k)·1; cached cost = w + k·r; caching wins when w + k·r < 1 + k, i.e. k > (w − 1)/(1 − r) reads.
• Equivalently, in total calls n = 1 + k: n > (w − r)/(1 − r). Both forms are the same rule; state which count you are using.
• Take w and r for the specific model and cache TTL from the provider's pricing page at use time — they differ by provider, model, and TTL. Every expiry or prefix change costs another write, so count only reads inside one TTL window.

The payoff depends on cache terms and prompt layout; use ai-prompt-engineering: Cache-Aware Prompt Layout to keep reusable prefixes stable.

Apply the same break-even logic before adopting batch-API discounts (take the current discount from the provider's batch pricing) versus real-time calls: batch trades latency for cost, so it fits delay-tolerant, non-interactive work (bulk classification, offline enrichment, nightly reports) — it is not a substitute for interactive features, which need real-time calls.

AI UX Patterns

Pattern	Rule
Loading states	Show tokens as they arrive on streamed surfaces; show a progress state on buffered (high-stakes) surfaces
Regenerate + stop	Always provide both controls; regenerate replays stored tool results, stop cancels pending tool executions
Confidence	Label AI output; use qualifiers for uncertain responses; never present with same certainty as DB reads
Feedback	Thumbs up/down minimum; corrections more valuable; route both into eval pipelines
Graceful degradation	Core product must work when AI provider is down — cache, fallback, or queue
Attribution	Clearly label AI-generated content; Art. 50 EU AI Act requires disclosure for interactive systems
Undo / edit	Allow editing before AI output takes effect; require confirmation for destructive actions

Guardrails & Safety

Layer	Rule
Input	Enforce length limits; detect instruction-override patterns; sanitize before prompt injection
Output	Scan for PII (names, emails, SSNs); use the provider's moderation endpoint if it offers one, otherwise a classifier call to a model; add a dedicated vendor (Lakera-class) for adversarial-grade detection (see Build vs. Buy)
High-stakes	Route medical/legal/financial AI output through human review; track review latency
Audit logging	Log prompts + responses separately from app logs; mask PII; set retention policy
Rate limiting	Rate limit to prevent abuse (injection attempts, data extraction) — beyond cost control
Fail closed	When guardrails time out or fail, block the release and log; never pass unfiltered. On streamed surfaces this means stop and retract; surfaces that must never show unchecked text are buffered (see Streaming Architecture)
Agent config lint	Before enabling tools on a feature, run python3 scripts/check_injection_defenses.py <agent-config.json>: it fails (exit 1) when the tool allowlist, retrieval-source allowlist, structured output validation or system-prompt isolation is missing

Multi-Provider Strategy

Abstract provider calls behind a single AIProvider interface (AI SDK's provider pattern achieves this). Never swap models for all users at once — use feature flags and circuit breakers.

Steps: provider abstraction (resolve current model IDs at each provider's docs at use-time, never hardcode a "best model"), fallback chain (primary → fallback → degraded mode) with a circuit breaker after N failures in M seconds, per-model prompt variants tested before switching traffic, and percentage A/B rollout gated on feedback, task completion, and error rates. See references/rollout-and-observability.md for the step table and eval loop patterns.

Fallback Equivalence Gate

Do not treat API compatibility as behavioral equivalence. Before enabling a fallback, replay a representative eval set against every provider/model path and compare schema validity, tool permissions, citation requirements, refusal behavior, latency, and unit cost. For cross-cloud failover, test an allowed same-model route before switching provider families, and verify residency, key custody, prompt/log retention, and outage independence. Do not splice a second provider into a partially delivered stream; start a new request with stored tool results and idempotency keys. If any required contract fails, use a named degraded experience.

Traps and Anti-Patterns

One list for the traps, the old Do/Avoid table and the architecture anti-patterns.

Trap	Do instead
Treating provider uptime as guaranteed, or graceful degradation as a diagram-only requirement	Keep the core product working when AI is down, and prove the fallback path under a real provider failure (see Fallback Equivalence Gate)
Keeping conversation history only in browser or mobile client state	Store history server-side as the canonical record for regeneration, resume and support workflows
Assuming schema-shaped output is safe because the model usually behaves, or parsing prose with regex	Use schema-constrained output or tool calling (choice rule in ai-prompt-engineering core-patterns) and validate every response before persistence or side effects
Swapping models or providers for all users behind the same endpoint	Re-run prompt, latency and eval gates, then roll out behind flags with A/B gates
Logging full prompts and responses without a PII and retention policy	Mask PII and set retention (see Guardrails: Audit logging)
Shipping AI output without user controls or labeling	Provide regenerate, stop generating and undo on every AI output; label AI-generated content for users and audit trails
Testing only against mocks, or only against live calls	Use mocked LLM responses in unit tests and real calls in integration tests
Streaming high-stakes output before the output guardrail passes	Match streaming mode to guardrail risk; buffer until the guardrail passes where unchecked text must never show (see Streaming Architecture)
Regenerate or stream-retry re-executing side-effecting tool calls	Idempotency keys per (conversation, turn, call index); replay stored tool results (see Conversation & Context Management)
Letting the model call or own the core product workflow	The product owns the workflow; otherwise the model becomes a single point of failure and a hard-to-audit orchestrator
Treating tool outputs as instructions	Tool results are an indirect prompt injection vector: treat them as untrusted data and parse and validate them before acting, especially before write or send operations
Hiding weak application contracts behind longer prompts	Fix validation, state and orchestration in code; prompt length hides these defects and makes them harder to debug
Letting the AI path directly mutate durable product state	Require confirmation, undo or compensating logic; the server owns writes, the model only requests them
Expanding one AI service into a catch-all abstraction	Keep routing, prompt logic, persistence, moderation and analytics under separate owners; a catch-all service also makes the injection attack surface unbounded

Scenarios

The end-to-end flow diagram and step-by-step recipes (streaming chat with citations, schema extraction with retry, injection-guarded tool calling, multi-provider routing on rate-limit, RAG with stale-cache invalidation) live in references/integration-scenarios.md.

Navigation

References

• references/product-integration-patterns.md — streaming UX, structured output, persistence, and degraded-mode patterns
• references/rollout-and-observability.md — cost controls, feature flags, evaluation loops, and operations
• references/integration-scenarios.md — end-to-end flow and step-by-step integration recipes (S1–S5)
• references/prompt-injection-and-ai-act.md — architecture-level injection defenses (the attack taxonomy is owned by ai-prompt-engineering), prompt/response logging, model version pin strategy, and a deployer-facing EU AI Act engineering checklist (disclosure, marking, logging) that points to qualified EU regulatory counsel for statute detail
• data/sources.json — official SDK, provider, and safety/eval sources

Related Skills

• software-backend — Backend service patterns
• software-frontend — Frontend application development
• software-realtime — Real-time communication and streaming
• software-security-appsec — Application security and threat modeling

Freshness Protocol

AI integration tooling changes rapidly. Freshness-check before answering questions about SDKs, model capabilities, or provider-specific patterns.

Triggers: SDK version questions, "is X still recommended?", streaming API changes, provider capability changes, new model releases.

Process: start from data/sources.json, run a targeted web search, check the changelog and migration guide of each SDK you use; app-level AI SDKs and provider SDKs release often, with breaking renames.

Regulatory Traps

Statute detail, status, and application dates are owned by qualified EU regulatory counsel. The engineering trigger table (prohibited practices, Art. 50 transparency, Annex III high-risk deployer duties, GPAI provider-vs-deployer split, enforcement) lives in references/prompt-injection-and-ai-act.md; confirm scope and dates with legal before relying on any row.

Indirect prompt injection is the primary exploit path: attacker-controlled data in retrieved documents, tool outputs, or web results overrides system instructions. Mitigations: isolate retrieved content with structural tags, enforce least-privilege tool scopes, validate model output before write/send operations, and test with adversarial documents in CI. Defense must be architectural — model-side mitigations reduce but do not eliminate risk.

See references/prompt-injection-and-ai-act.md for injection defence patterns and the deployer logging checklist.

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.