Prompt-History Inspect — First Move Before Anything Else
"The prompt you wrote is not the prompt the model received."
— Operating axiom for every framework that templates, injects few-shots, appends tool definitions, or wraps system messages.
1. 何时激活 (When to activate)
Activate this skill the moment an LM call surprises you, BEFORE any other debug move.
| Trigger | Signal |
|---|
| Output wrong | "Why did it answer X?" / hallucinated fact / wrong format / refusal |
| Output truncated | mid-sentence cut, partial JSON, missing fields |
| Output empty / repeats | model returns "", repeats the same token, loops |
| Behaviour changed | "It worked yesterday" / "It worked on GPT-4o but not on Llama-3" |
| Cost / latency spike | tokens jumped 3× without code change → something got injected |
| Tool call wrong | wrong tool picked, args malformed, tool call missing |
| Schema validation failed | Pydantic / Outlines / guidance grammar refused output |
| Eval regression | metric dropped after upgrading framework version |
| Production bug | a user-facing thread produced a wrong answer — need to see what the LM saw |
Do NOT activate when:
- You are authoring a new prompt for the first time (no rendered prompt exists yet).
- The bug is clearly outside the LM call (retriever returned empty, API key invalid, network down).
- The framework hasn't even been called yet (e.g., import error, schema validation pre-call).
The trigger is universal across the stack. Any framework that templates a prompt — DSPy, LangChain, LangGraph, CrewAI, LlamaIndex, Aider, Guidance, Outlines — has a layer between "what you wrote" and "what the model received." This skill is the first-line probe into that gap.
2. 核心心智模型 (Core mental model)
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ What you wrote │ ≠ │ What was rendered│ ≠ │ What the LM saw │
│ (template, │ │ (after few-shot │ │ (after provider │
│ signature, etc) │ │ injection, tool │ │ reformatting, │
│ │ │ defs appended, │ │ message squash, │
│ │ │ system msg, │ │ token truncation)
│ │ │ history, etc) │ │ │
└──────────────────┘ └──────────────────┘ └──────────────────┘
layer 1 layer 2 layer 3
(your code) (framework render) (provider transport)
Three mental shifts the agent must internalize:
-
The rendered prompt is the ground truth, not your source code. Frameworks silently inject system messages, append tool JSON-schemas, deduplicate messages, summarise history, truncate context, reorder fields. The only trustworthy artefact is what was sent on the wire.
-
Inspect first, change nothing. Premature prompt edits hide the bug. If you "fix" the prompt before seeing the rendered output, you're optimising against a hallucination of the problem. The discipline is: dump → diff against expectation → identify which layer diverged → fix at that layer.
-
The inspect command is framework-specific but the SOP is universal. DSPy gives you inspect_history(n=1). LangChain gives you set_debug(True). Raw SDKs give you OPENAI_LOG=debug or HTTPX event hooks. You learn one cheat sheet (§7) once and the SOP applies everywhere.
The 30-second test. Before you do anything else, you should be able to print the exact final prompt text in <30 seconds. If you can't — you're not set up to debug LMs. Fix that first.
3. SOP 工作流 (SOP workflow)
A four-step ritual. Do them in order. Do not skip ahead.
Step 1 — Dump the rendered prompt (≤30 sec)
Look up the framework on the §7 cheat sheet. Run the inspect command. Get the actual text of:
- The system message (if any)
- The user message
- Any few-shot demos that were injected
- Tool / function definitions (if any)
- Assistant prefill (if any)
- The model's raw response
For DSPy: dspy.inspect_history(n=1) [dspy.ai/api/utils/inspect_history/].
For LangChain: from langchain.globals import set_debug; set_debug(True) [python.langchain.com/api_reference/core/globals/langchain_core.globals.set_debug.html].
For raw SDKs: export OPENAI_LOG=debug or ANTHROPIC_LOG=debug [github.com/openai/openai-python, github.com/anthropics/anthropic-sdk-python].
Step 2 — Diff against expectation
Write down (mentally or in a scratch file): what did I expect the prompt to contain? Then compare to the dump. Look for:
- Extra content you didn't intend (auto-injected system message, hidden few-shot, tool JSON-schema bloat).
- Missing content you did intend (your variable interpolation rendered as
None / empty / {var} literal).
- Wrong order (assistant message before system, tool call before tool result).
- Truncation (your 10k-token context was clipped to 4k; few-shot ate your real input).
- Encoding (raw HTML/JSON characters un-escaped, leaking template markers like
{% if %}).
Step 3 — Identify the divergence layer
Now classify the bug into one of three layers (mental model §2):
- Layer 1 (your code): variable not interpolated, wrong key passed to chain, signature field misnamed → fix in your code.
- Layer 2 (framework render): framework's template logic injected something unwanted, demos chosen by optimizer don't match task, tool definitions inflated past budget → fix at the framework config (different prompt template, fewer demos, smaller tool set, different module).
- Layer 3 (provider transport): provider's message squashing, role coercion, token cap, finish_reason="length" → fix at provider config (different model, raise
max_tokens, restructure to avoid role coercion).
Step 4 — Fix at the correct layer, then re-dump
Apply the minimum change at the identified layer. Re-run. Re-dump. Confirm the prompt now matches expectation. Then check whether the bug is fixed.
Critical anti-pattern: do not skip step 4's re-dump. Many "fixes" change Layer 1 when the real bug is Layer 2 — the prompt still looks broken on re-dump, even if the visible symptom changed.
4. 操作模型 (Operational model — Trigger / Action / Output / Evidence)
OP-1 · Inspect last N DSPy LM calls
- Trigger: A DSPy
Predict / ChainOfThought / ReAct returned wrong output, OR a compile run hung mid-trial.
- Action:
import dspy; dspy.inspect_history(n=1) — increase n to see the last few calls. For per-LM history: lm.inspect_history(n=3).
- Output: Prints system / user / assistant blocks plus the raw response for the last N LM invocations.
- Evidence: dspy.ai/api/utils/inspect_history/, DSPy docs "Debugging & Observability" [dspy.ai/tutorials/observability/]. Note:
inspect_history shows only LM calls, not retriever/tool calls — for those, wire mlflow.dspy.autolog() [github.com/stanfordnlp/dspy issue #784 for n-parameter quirks].
OP-2 · LangChain global debug / verbose
OP-3 · LangGraph time-travel state inspection
- Trigger: A LangGraph thread produced a wrong final answer; you need to see the state (and the LM input) at step k, possibly fork from there.
- Action:
history = list(graph.get_state_history(config)) # newest → oldest
for snap in history:
print(snap.config["configurable"]["checkpoint_id"], snap.values)
# Replay from a checkpoint:
graph.invoke(None, config={"configurable": {
"thread_id": "...", "checkpoint_id": "<chosen id>"}})
# Fork by modifying state:
graph.update_state(config, {"some_field": "new value"})
- Output: Full per-step state list (including messages, the inputs the LM saw); ability to replay or fork.
- Evidence: langchain-ai.github.io/langgraph/concepts/time-travel/, docs.langchain.com/oss/python/langgraph/use-time-travel. Caveat: replay re-executes nodes — LLM calls and API requests fire again and may produce different results [time-travel docs]. To inspect without re-cost, read state from history only; do not invoke.
OP-4 · CrewAI step-by-step inspection
- Trigger: A CrewAI agent picked the wrong tool, looped, or delivered the wrong output; you need per-step visibility.
- Action: Two layers:
Agent(..., verbose=True) # prints agent thoughts + tool I/O
Crew(..., verbose=True)
# Per-step structured capture:
def my_step(step_output):
print("STEP:", step_output) # AgentAction / AgentFinish / observation
Agent(..., step_callback=my_step)
- Output: Every agent action, tool input, tool observation, and final answer printed or captured to your callback.
- Evidence: docs.crewai.com/en/concepts/agents (
step_callback parameter), docs.crewai.com/en/observability/overview. CrewAI's internal log is thin — for production observability also wire mlflow.crewai.autolog() or Langtrace [docs.crewai.com/how-to/langtrace-observability].
OP-5 · Aider show last diff and full prompts
OP-6 · Raw OpenAI / Anthropic SDK debug logging
OP-7 · HTTPX event-hook logger (full control)
- Trigger: You need the full raw HTTP request body (e.g., to verify what
additional_drop_params or extra_body did), and OPENAI_LOG=debug is too noisy or formats badly.
- Action: Pass a custom
httpx.Client with event hooks to the SDK:
import httpx, json
from openai import OpenAI
def log_request(req):
print("REQUEST:", req.method, req.url)
print(json.dumps(json.loads(req.content), indent=2))
client = OpenAI(http_client=httpx.Client(event_hooks={"request": [log_request]}))
- Output: Clean, structured dump of the exact JSON body sent over the wire.
- Evidence: Simon Willison's TIL [til.simonwillison.net/httpx/openai-log-requests-responses], httpx event-hooks docs [encode/httpx discussion #3073]. Works for both OpenAI and Anthropic SDKs (both accept
http_client=).
OP-8 · Tool-definition bloat probe
- Trigger: Tokens / latency spiked, model "got dumber" after adding tools, or context-window error fires before your real input.
- Action: Dump the prompt (any of OP-1–7), then count tokens in the tool-definitions block alone. Most frameworks serialise each tool's JSON-schema (description, parameters, types, enums) into the prompt — 10 tools easily costs 2k+ tokens before any user input.
- Output: Concrete token cost per tool; identification of which tools to drop or compress.
- Evidence: Reproducible by inspect-dump; community pattern documented in agent-framework debugging threads (e.g., github.com/microsoft/semantic-kernel discussion #1239 on getting the raw prompt).
OP-9 · Production trace inspection (LangSmith / LangFuse / MLflow / Phoenix)
- Trigger: Bug surfaced in production; you don't have a repro locally; you need to inspect a real user's thread.
- Action: Open the trace in the configured platform (LangSmith / LangFuse / Phoenix / MLflow), navigate to the failing run, click into the LM call, view the rendered prompt and response. This is the production-equivalent of OP-1–7.
- Output: Reviewed prompt for the failing run, ready for diff against expectation (Step 2 of SOP).
- Evidence: Use the corresponding skill (
langsmith, phoenix, langfuse) for the platform-specific UI. This skill enforces the first-move SOP; those skills provide the UI.
OP-10 · Save the rendered prompt as a fixture
- Trigger: You've found the bad rendered prompt and now need to reproduce / regression-test.
- Action: Save the dumped prompt text to
tests/fixtures/prompt-bug-<id>.txt. Add a test that asserts the next render does not contain the bad pattern. (For LangChain, snapshot the prompt.format(**inputs) output; for DSPy, snapshot what inspect_history printed.)
- Output: A regression test that pins the prompt rendering, not just the model output.
- Evidence: Same principle as snapshot testing in UI — the rendered prompt is the artefact to pin.
5. 困境决策案例 (Dilemma cases)
Case A — "Framework template silently truncated my context"
困境 (Dilemma): A RAG pipeline retrieves 10 passages and asks the LM to synthesize. Outputs miss obvious facts that are in the retrieved passages. User's first instinct: "the model is bad / the retriever is bad / let me re-rank."
约束 (Constraints):
- 10 passages, ~600 tokens each = ~6k tokens of context.
- Framework: a custom LangChain
MapReduceDocumentsChain wrapper.
- Model: GPT-4o, 128k context — no obvious context-budget issue.
决策步骤 (Decision steps):
- Refuse to re-rank, refuse to swap the model, refuse to "fix" the prompt. First move is inspect (§3 Step 1).
- Enable
set_debug(True) from langchain.globals [python.langchain.com/api_reference/core/globals/langchain_core.globals.set_debug.html]. Re-run.
- In the dumped prompt: notice the prompt only contains 3 passages, not 10. The framework's map-reduce splits and the reduce prompt got only the per-chunk summaries — and the chunk-summarizer dropped facts.
- Layer classification (§3 Step 3): Layer 2 — framework render. The bug is the map-reduce decomposition, not the model or the retriever.
- Fix: switch to a
stuff chain (all 10 passages in one prompt — fits in 128k) or write better chunk-summary prompts. Re-dump to confirm all 10 passages now appear.
结果 (Outcome): Wrong layer would have been: a week of re-ranker tuning. Right layer: 10 minutes of template fix. Visible only via rendered-prompt dump.
可提取的操作 (Extractable operation): When a RAG pipeline misses obvious retrieved facts, dump the prompt and count how many retrieved chunks actually appear. The framework probably dropped some.
Case B — "Tool definitions inflated my prompt past budget"
困境: A LangChain or CrewAI agent worked fine with 3 tools. After adding 4 more tools, accuracy dropped and latency tripled. User suspects the model "gets confused by more tools."
约束:
- 7 tools, each with rich JSON-schema (descriptions, enum params, nested objects).
- Model: claude-3.5-sonnet, 200k context.
- The new tools are individually correct (unit tests pass).
决策步骤:
- First move: dump the prompt. OP-7 (HTTPX hook) gives the cleanest view of the raw JSON body. Count tokens in the
tools=[...] array alone (OP-8).
- Finding: tool definitions now consume 3.4k tokens — they replaced ~3k of the available reasoning budget. The system message + few-shots + tools eats the head of the context, pushing user input + history toward the tail where attention is weaker.
- Layer classification: Layer 2 (framework render) AND Layer 3 (provider transport — attention degrades in long-context tails for many models).
- Fix options:
- Compress descriptions (the cheapest tool description halved tokens with no quality loss).
- Lazy-load tools (route first, expose only relevant tools per query).
- Drop verbose
enum lists in params — describe them in natural language.
- Re-dump to confirm token count dropped. Re-test.
结果: Without inspect, the user would "fix" by removing tools (losing capability) or switching models (expensive). Inspect reveals the tool-schema is the cost driver.
可提取的操作: More tools = silent prompt inflation. Always inspect tool-block token count before blaming the model.
Case C — "Optimized DSPy program degraded after model swap — why?"
困境: A MIPROv2-compiled program for GPT-4o hits 85% on dev. Re-pointed at Llama-3-8B, drops to 41%. User assumes Llama is just weaker.
约束:
compiled.json saved with demos + instructions tuned to GPT-4o.
- Same program code, only
dspy.configure(lm=...) changed.
- Cost: re-compile is $2–5 vs. accepting the 41% loss.
决策步骤:
- First move (per this skill, not DSPy doctrine yet): dump.
dspy.inspect_history(n=3) on Llama-3-8B [dspy.ai/api/utils/inspect_history/].
- Finding: the bootstrapped demos in
compiled.json contain verbose, GPT-4o-style chains-of-thought (5–8 sentences per demo). Llama-3-8B copies the length but skips the reasoning structure — producing plausible-shaped but wrong outputs.
- Layer classification: Layer 2 — the rendered prompt looks fine for GPT-4o, wrong for Llama-8B. The artefact is model-coupled.
- Fix per DSPy doctrine [Case B of
dspy-sop]: recompile against Llama-3-8B. But without the inspect step, the user would not have known the demos were the bottleneck (vs. the instructions, or the signature).
- Optional: after re-compile, re-dump to confirm the new demos are shorter / more explicit.
结果: Inspect reveals what changed in the rendered prompt; doctrine says what to do about it. Skipping inspect leads to "Llama is bad" — wrong root cause.
可提取的操作: Compiled-prompt artefacts are model-coupled. Inspect-dump on the new model is mandatory before declaring the model "weaker."
Case D — "Production thread gave wrong answer — can't reproduce locally"
困境: A LangGraph customer-support agent gave a confidently wrong answer in production. User logs show only the final output, not intermediate state. No local repro.
约束:
- Production has a checkpointer configured (Postgres-backed).
- The thread ID is known.
- Can't replay full LLM calls (production cost / rate-limit risk).
决策步骤:
- First move: inspect the production trace. Use OP-3 with the production checkpointer:
history = list(graph.get_state_history({"configurable": {"thread_id": "<prod-id>"}}))
- Walk back from the final state to find the LM call that introduced the error. View the
messages field in each StateSnapshot.values — that's the rendered prompt as seen by the LM [langchain-ai.github.io/langgraph/concepts/time-travel/].
- Finding: at step 4, a tool returned an error string that the next node concatenated into the system message — confusing the LM into a confabulated answer.
- Layer: Layer 1 (your code) — node didn't error-handle the tool result.
- Do not call
graph.invoke(None, config=...) to "replay" — that re-executes LM calls and incurs cost [time-travel docs caveat]. Reading state history is read-only and free.
- Fix the node's error-handling, push, write a regression test using the captured state as fixture (OP-10).
结果: Time-travel reads state without re-paying for LLM calls. The bug is visible in the inspected state, not in the final output alone.
可提取的操作: For production bugs, get_state_history is read-only and free; invoke(None, config=...) is replay and costs LM calls. Inspect first, replay only if necessary.
6. 反模式与边界 (Anti-patterns & boundaries)
Anti-patterns (in rough order of how often they cost engineer-hours)
- Changing the prompt before inspecting it. This is the #1 mistake. If you don't know what was sent, you don't know what to change. Inspect costs 30 seconds; "fix the prompt and see" costs an afternoon.
- Trusting the source code matches what was sent. False every time there's a templating, few-shot, signature, or tool-injection layer between your code and the provider. Treat source as a hypothesis; the dump is evidence.
- Inspecting at the wrong layer. Dumping a
PromptTemplate.format(...) output is not the same as dumping what hit the wire — the framework adds messages, system instructions, tool schemas after that point. Prefer SDK-level (OPENAI_LOG=debug) or HTTPX-hook (OP-7) over template-render for production debugging.
- Leaving
set_debug(True) / OPENAI_LOG=debug on in production. Both leak request bodies, and OPENAI_LOG=debug / ANTHROPIC_LOG=debug print the API key in plaintext [openai-python issue #1196]. Always scope to debug sessions; toggle off when done.
- Replaying production threads via
graph.invoke(None, config=...) to debug. Re-executes LM calls and tools, paying real cost, possibly mutating real systems. Read state history; replay only when needed [time-travel docs].
- Inspecting only the prompt, ignoring the response. Many bugs (model truncation, refusal, schema validation) are visible only in the raw response including finish_reason. Always dump both.
- Using
inspect_history as the only observability. DSPy's inspect_history shows LM calls only — not retrievers, not tools, not subgraphs. For multi-component pipelines, layer it with MLflow / LangSmith tracing [dspy.ai/tutorials/observability/].
- Eyeballing the dump. For long prompts, diff against a known-good rendering rather than reading top-to-bottom. Token-count the segments.
- Reasoning about why the prompt is "fine" without re-rendering after each change. Frameworks change prompts at runtime based on history, retries, tool state. Each invocation can produce a different rendered prompt. Re-dump.
- Sharing a debug log without scrubbing keys. Strip
Authorization: headers before pasting into chat / issues / Slack.
Boundaries (when this skill is NOT the right move)
- Pre-call errors (import error, auth failure, network down) — no LM call happened; there's nothing to inspect. Fix infra first.
- Prompt authoring from scratch — no rendered prompt exists yet. Use prompt-engineering / DSPy-Signature / template-design skills instead.
- Eval-time aggregate metrics — single prompt dump is the wrong granularity. Use eval / langsmith / phoenix.
- Streaming-only debugging — if you only care about token-by-token streaming behaviour, framework streaming hooks are better than post-hoc inspect.
- You already have a deep trace UI configured (LangSmith, LangFuse, Phoenix, MLflow) — the first move SOP still applies (OP-9), but the implementation is "open the trace" not "run inspect_history". Same skill, different action.
7. 跨框架对照 (Cross-framework cheat sheet)
Decision tree (one screen)
LM call surprised you?
│
Yes ──► STOP. Do not edit the prompt. Do not retry. Do not swap models.
│
▼
What framework?
DSPy → dspy.inspect_history(n=1)
LangChain → set_debug(True)
LangGraph → graph.get_state_history(config)
CrewAI → step_callback + verbose=True
Aider → /diff AND aider --verbose
Raw OpenAI/Anth → OPENAI_LOG=debug / ANTHROPIC_LOG=debug
Clean dump → httpx event hook (OP-7)
Production bug → LangSmith / LangFuse / Phoenix trace
│
▼
Diff dump against expectation. Identify layer (1/2/3 — §3 Step 3).
│
▼
Fix at correct layer. Re-dump. Confirm prompt now matches.
│
▼
NOW check if the bug is fixed. If not, repeat from inspect.
Ecosystem note
This skill is adjacent to but distinct from trace-UI skills:
langsmith, phoenix, langfuse, mlflow provide the UI for inspecting traces.
prompt-history-inspect provides the SOP (inspect-first discipline) and the cross-framework command lookup.
Use them together: this skill says when and why to inspect; the platform skills say where the UI lives. For local dev, the framework-native commands in §7 are usually enough.
Key citations (all official)