Agent Metric Dashboards

The five agent KPIs: sessions and turns per session, deflection, latency percentiles, reasoning intensity (the observable cost driver), and quality. This skill wires each to a real source and lays out the single-pane dashboard an executive reviewer can read without being misled.

The Prerequisite That Gates Everything

Agentforce Session Tracing must be enabled (Setup, Einstein Audit, Analytics, and Monitoring Setup, Agentforce Session Tracing) before the conversations you want to measure. UNVERIFIED (2026-10-03): the earlier statements, sourced to Help articles that do not fetch, that analytics never backfill and that readers need the "Data Cloud User" permission set; plan as if both are true. Agent Analytics runs on Data Cloud, consumes credits, and is not available on Data Cloud One companion orgs (Generative AI guide, Agentforce Analytics).

This makes dashboard enablement a row on the go-live checklist (agentforce/agent-deployment-checklist), not an analytics backlog item.

Questions to Ask Before Configuring

Ask these before building a tile. Each one traces to a gotcha in references/gotchas.md.

Question	Why it matters	What a good answer adds	What proper configuration adds over just doing it
"When was session tracing turned on, and is it on in every org that matters?"	The dashboard has no history before tracing (Gotcha 1)	A data start date and a go-live checklist row	A trend nobody misreads as a launch date
"Is there a control arm, or only a before-and-after?"	Deflection without a holdout measures the world (Gotchas 3, 4, 5)	A holdout design or an honest tile label	Causal claims only where they are earned
"What does cost mean here: tokens, credits or currency?"	Token counts are queryable from the usage DMO; currency is not (Gotcha 7)	Driver tiles plus a monthly reconciliation with consumption reporting	A cost view that is true and actionable
"Who reads the dashboard, and do they have Data Cloud access and folder access?"	Readers without access see nothing (Gotcha 2)	A permission set group and a shared folder	A dashboard that works for its audience on day one
"Which events must annotate the trends?"	Model and version changes move metrics silently (Gotcha 8)	Agent activations, prompt activations, model changes as series	Step changes explained at a glance
"Who owns each alert, and how was its threshold chosen?"	Untuned alerts get muted (Gotcha 12)	An owner and a distribution-based threshold per alert	Alerts that are still read in month four

Where The Numbers Actually Come From

KPI	Source (DMO label, object API name)	Notes
Sessions	AI Agent Session, ssot__AiAgentSession__dlm	Also in the standard Total Session report
Turns per session	AI Agent Interaction, ssot__AiAgentInteraction__dlm ÷ sessions	ssot__TopicApiName__c gives the subagent per turn
Deflected and escalated sessions	ssot__AiAgentSessionEndType__c on the session (resolved, escalated, deflected, other)	Also the standard Deflection Session report
Average latency	Standard Latency report	Compute p50 and p95 from interaction timestamps yourself
Action failure rate, LLM calls per turn	AI Agent Interaction Step, ssot__AiAgentInteractionStep__dlm	Step types UserInputStep, LLMExecutionStep, FunctionStep; group by action, sort by rate
Tokens	Ai Agent Generative Ai Usage, AiAgentGenerativeAiUsage_std__dlm (Spring '26 and later)	Token counts per request; no currency
Quality, feedback	Session-tracing metric scores and feedback signals	Calibrate against human labels quarterly

There is no standard Conversation__c. A spec that names it is a data-engineering project, not a dashboard. Token counts are queryable from the usage DMO; a currency rate is not in org data (this corrects the earlier "no queryable per-token cost"). Object and field names come from the Data Cloud DMO reference; references/metadata-examples.md has a deployable calculated insight and the driver queries.

Adoption Signals

Every production agent from activation onward; monthly executive review. Tracing must be on from day zero, so this skill is consumed before launch rather than after the first week.

Recommended Workflow

1. Enable Session Tracing and the Session Tracing Data Model before activation; assign Data Cloud User to every intended reader and verify with a real one.
2. Build the volume and efficiency tiles from ssot__AiAgentSession__dlm, ssot__AiAgentInteraction__dlm and ssot__AiAgentInteractionStep__dlm: sessions, turns per session, LLM calls per turn, p50 and p95 latency. Deploy the daily session counts as a calculated insight (references/metadata-examples.md). Split agent latency from action latency, different owners, different fixes.
3. Make deflection causal or rename it. Randomise a holdout arm at the routing layer and report (rate_control − rate_treatment) / rate_control with both arm sizes; where no holdout is possible, label the tile "Escalation rate (no control arm)".
4. Pair every proxy with its confound on the same tile: deflection with 72-hour repeat-contact rate, CSAT with n and response rate, aggregate with the per-subagent breakdown sorted by rate (subagent is the April 2026 rename of topic; the metadata names did not change).
5. Report cost as drivers (sessions × turns/session × LLM-calls/turn, plus tokens from AiAgentGenerativeAiUsage_std__dlm) and take absolute cost from Salesforce consumption reporting (Digital Wallet). Reconcile monthly; never compute a currency figure from org data alone.
6. Annotate every trend with agent version activations, prompt template activations, and model version changes — model versions move without a deployment on your side.
7. Derive alert thresholds from four weeks of observed distribution, give each one a named owner, and review at four weeks: did it fire, was every firing actionable.

Key Considerations

• Deflection rises when the agent gets worse — a user who abandons looks identical to one who was helped. It is never a solo KPI.
• Mean latency hides the tail that drives abandonment. Report p50 and p95.
• A quality score whose agreement with human judgement is unmeasured is an unmonitored dependency of every decision made from it.
• The OTel export (GET /services/data/v66.0/einstein/audit/otel/{session-id}) is one session per call, limited to the previous 72 hours, and Beta. Sample rather than exporting everything, and alert on exporter lag.

Worked Examples (see references/examples.md)

• Deflection with baseline — Service org with 40% pre-agent escalation rate.
• Tokens/conversation trend — Costs spike after a subagent-instruction rewrite.

Common Gotchas (see references/gotchas.md)

• CSAT response bias — Only frustrated users answer the survey — CSAT looks terrible.
• Deflection = 'user gave up' — No escalation because user closed the browser in frustration.
• Cost metric without model version — Cost/conversation changes overnight due to model upgrade.

Top LLM Anti-Patterns (full list in references/llm-anti-patterns.md)

• Single-number CSAT with no context.
• Deflection without a baseline — reports vanity metrics.
• LLM-as-judge never calibrated — grades itself.

Official Sources Used

See references/well-architected.md for the sources read for this revision.