langfuse-tracing

Overview

Langfuse is a production LLM observability platform (per lf-gh).

Step 1 - Install

Per lf-gh:

pip install langfuse

For TypeScript:

npm install @langfuse/tracing

Set up project credentials per Langfuse self-hosted or cloud project (LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY + LANGFUSE_HOST).

Step 2 - Instrument with @observe

Per langfuse.com/docs/sdk/python/decorators:

Python:

from langfuse import observe

@observe(name="llm-call", as_type="generation")  # auto-captures inputs, outputs, timings, errors
async def my_async_llm_call(prompt_text):
    return "LLM response"

TypeScript - create an observation, do work inside it, then end it:

import { startActiveObservation } from "@langfuse/tracing";

const { observation, end } = startActiveObservation({
  name: "llm-call",
  type: "generation",
  input: { prompt: promptText },
});

try {
  const result = await callMyLLM(promptText);
  observation.update({ output: result });
  return result;
} finally {
  end();
}

Validation: After your first instrumented call, open the Langfuse UI → Traces. You should see a new trace with nested observations, inputs, outputs, and timings. If no trace appears within ~30 s:

• Confirm LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST are set in the process environment.
• Call await langfuse.flushAsync() (TS) / langfuse.flush() (Python) before process exit to force-drain the queue.
• Check langfuse.debug() output for connection errors.

Step 3 - Update current observation with metadata

Per lf-py-deco:

from langfuse import get_client

langfuse = get_client()

with langfuse.start_as_current_observation(as_type="span", name="data-processing"):
    langfuse.update_current_span(metadata={"step1_complete": True})

Common metadata fields used in production:

• model - the model name (e.g., claude-haiku-4-5)
• model_parameters - temperature / top_p / max_tokens
• usage - input/output tokens, cost
• tags - environment (prod / staging), feature flag, customer ID
• level - DEBUG / DEFAULT / WARNING / ERROR

Step 4 - Score traces

Per langfuse.com/docs/scores:

langfuse.score(
    trace_id="...",
    name="answer_relevance",
    value=0.87,           # numeric (0-1); also supports categorical (string) and boolean
    comment="Judged by GPT-4 rubric"
)

Scores can come from:

• Manual review (human raters via the Langfuse UI)
• Automated eval (run a metric in batch, score by trace_id)
• User feedback (thumbs-up / thumbs-down from the app)

Validation: After scoring, open the Langfuse UI → Traces → select the trace. The score should appear in the Scores panel. If missing, confirm the trace_id matches an existing trace and that the score call did not raise an exception.

Step 5 - Datasets for offline eval

Datasets are collections of (input, expected_output) items built from production traces, CSV / JSONL imports, or the UI. Run one against your app to diff vs a baseline:

items = langfuse.get_dataset_items(dataset_id="...")
for item in items:
    actual = my_llm_app(item.input)
    item.run(actual)  # links the run back to the dataset for diff vs baseline

See references/datasets.md for building datasets from production traces, the lf-ds API signature, and the run-validation checkpoint.

Step 6 - Prompt management

Fetch the current production prompt at runtime and pin its version so prompt drift is traceable:

from langfuse import get_client

langfuse = get_client()

# Fetch the production-labelled prompt; falls back to cache if offline
prompt = langfuse.get_prompt("my-prompt-name", label="production")
compiled = prompt.compile(user_input=user_query)

# Pass compiled text to your LLM call
response = my_llm_call(compiled)

Iterate prompt text in the Langfuse UI and roll out new versions per environment (production / staging labels) without code deploys. Version history and A/B comparison are available in the UI.

Validation: After an instrumented call using a managed prompt, open the Langfuse UI → Traces → select the trace. The prompt name and version should appear in the trace metadata, confirming prompt version is pinned and attributable.

Step 7 - CI integration

Langfuse is observability-side, not pre-deploy CI-side. Post-deploy CI patterns query the Langfuse API for recent traces and assert on aggregate metrics (a score-query gate, eval-on-trace, cost-regression, score-based alerting). Runnable gate script and the full pattern list: references/ci-integration.md.

Anti-patterns

Anti-pattern	Why it fails	Fix
Trace everything in production with no sampling	Cost explodes at scale	Use level=DEBUG + UI-side sampling (Step 3)
Score traces only via UI (no automated path)	Can't catch silent regressions	Automated langfuse.score() per trace (Step 4)
Pull production trace inputs without privacy review	PII leakage into eval datasets	Cross-ref synthetic-pii-generator for fixture sanitization before promotion
Skip prompt versioning	Prompt drift breaks attribution	langfuse.get_prompt() with version pin (Step 6)
Conflate Langfuse with pre-deploy eval	Tries to be both; wins neither	Pair Langfuse (post-deploy) with Promptfoo/DeepEval/Ragas (pre-deploy)

Version notes

Per lf-gh, the SDK was rewritten in v4 and released in March 2026; this skill targets the v4 API. v3 patterns are no longer supported - pin the SDK version in requirements, and for v3 codebases see the upstream migration guide.

Limitations

• Langfuse cloud is hosted; for data-residency-strict teams, self-host (well-supported but operational overhead).
• SDK and score-API signatures may drift between versions; always consult langfuse.com/docs and lf-scores when authoring wiring against any rapidly-evolving endpoint.
• Integration with eval frameworks evolves rapidly; consult langfuse.com/docs for current Promptfoo / DeepEval / Ragas integration patterns.

References

• lf-gh - Python SDK repo, install, version note
• lf-py-deco - @observe decorator + observation patterns
• lf-scores - score API
• lf-ds - datasets
• lf-docs - full documentation
• promptfoo-evaluation, deepeval-evaluation, ragas-evaluation, giskard-llm - pre-deploy eval sister tools