Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.
Build and run evaluators for AI/LLM applications using Phoenix.
Review the instructions the agent will receive before you install.
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
Production: production-overview → production-guardrails → production-continuous
| Prefix | Description |
|---|---|
fundamentals-* | Types, scores, anti-patterns |
observe-* | Tracing, sampling |
error-analysis-* | Finding failures |
axial-coding-* | Categorizing failures |
evaluators-* | Code, LLM, RAG evaluators |
experiments-* | Datasets, running experiments |
validation-* | Validating evaluator accuracy against human labels |
production-* | CI/CD, monitoring |
| Principle | Action |
|---|---|
| Error analysis first | Can't automate what you haven't observed |
| Custom > generic | Build from your failures |
| Code first | Deterministic before LLM |
| Validate judges | >80% TPR/TNR |
| Binary > Likert | Pass/fail, not 1-5 |
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Map an existing codebase into seven documents covering its technology, structure, architecture, conventions, integrations, tests, and concerns. Link findings to file evidence, mark unknowns, and highlight differences between stated plans and the actual implementation.
Assess a repository’s readiness for AI agents with AgentRC and generate a standalone HTML dashboard. Review scores across nine areas, see prioritized improvements, and optionally apply scoring policies tailored to an organization.
Generate coding agent instruction files from a project's actual code using AgentRC. Capture detected languages, frameworks, and conventions in one shared file or split rules by topic and project area, with a preview before writing.
Customize AgentRC readiness assessments to match team or organization standards. Choose or create policies that adjust check importance, disable irrelevant checks, set pass thresholds, and define requirements for automated checks in CI workflows.
Analyze supplied ad performance data to identify wasted spend, strong campaigns, and next tests. Compare channels, assess whether test differences are meaningful, and produce budget reallocation recommendations with a prioritized action plan.
Turn code files into learning resources by adding comments that explain syntax, programming concepts, and design choices. Adjust explanation depth and repetition to the reader’s knowledge while preserving file structure, encoding, and syntax.