ORC Evaluation (LLM-as-Judge)

Set up automated quality evaluation for ORC workflow outputs. Read docs/EVALUATION-COMPONENT.md for the full reference.

Require

(require '[ai.obney.orc.evaluation.interface :as eval])

Overview

The evaluation component provides LLM-as-judge scoring:

• Grounding — Is the output faithful to the input?
• Instruction-following — Does the output follow the instruction?
• Reasoning — Is the reasoning sound?
• Completeness — Does the output address all aspects?

Each judge returns a score (0.0-1.0) with structured feedback.

Built-in Judges

Evaluate one trace

(eval/evaluate-trace
  {:instruction "Summarize the article covering all main points"
   :inputs {:article "Long article text..."}
   :outputs {:summary "Short summary..."}}
  {:judges [:grounding :completeness]})
;; => ScoreWithFeedback with :score, :feedback, :dimensions, and—when
;;    model-backed—durable :model-provenance on the recorded score event

Workflow Judges (DSL Integration)

Attach judges to workflow nodes for automatic scoring during execution:

(orc/workflow "evaluated-pipeline"
  (orc/blackboard {:question :string :answer :string})

  ;; Define judge configurations
  (orc/judges
    {:grounding    {:type :grounding :weight 0.5}
     :completeness {:type :completeness :weight 0.5}})

  ;; Attach judges to a node
  (orc/llm "answer"
    :instruction "Answer the question thoroughly."
    :reads [:question]
    :writes [:answer]
    :judges [:grounding :completeness]))

When executed with tracing enabled, each judged node produces evaluation scores in the trace.

Batch Evaluation

Evaluate a workflow across multiple test cases:

(eval/evaluate-traces
  [{:instruction "Answer the question"
    :inputs {:question "What is AI?"}
    :outputs {:answer "Artificial intelligence..."}}
   {:instruction "Answer the question"
    :inputs {:question "What is ML?"}
    :outputs {:answer "Machine learning..."}}]
  {:judges [:grounding :completeness]})
;; => {:avg-score 0.82 :results [...] :min-score ... :max-score ...}

GEPA Integration

Evaluation judges feed directly into GEPA optimization. When GEPA proposes instruction candidates, judges score them — the scores drive Pareto selection.

;; GEPA uses judges automatically when configured
(gepa/optimize! ctx
  {:sheet-id sheet-id
   :node-name "answer"
   :judges {:grounding 0.5 :completeness 0.5}
   ...})

Reference

• docs/EVALUATION-COMPONENT.md — Full evaluation guide
• docs/SELF-IMPROVING-LOOP.md — Evaluation and continuous improvement
• docs/GEPA-GUIDE.md — How judges integrate with optimization