RL Reward Construction with OpenJudge
Build reward signals for reinforcement learning from human feedback (RLHF) and
reinforcement learning from AI feedback (RLAIF) using the openjudge library.
When to Use This Skill
- Building scalar rewards for GRPO / REINFORCE rollout scoring
- Generating (chosen, rejected) preference pairs for DPO / IPO
- Best-of-N candidate selection
- Multi-dimensional reward shaping (correctness + safety + format)
- Replacing or bootstrapping a reward model with LLM-as-judge
Step 1 — Choose Your Reward Strategy
Use this decision tree before writing any code:
RL Algorithm + Task type?
│
├── GRPO / REINFORCE — Verifiable task (math, code, structured output)
│ └── → POINTWISE ✅ (FunctionGrader, exact score, zero LLM cost)
│
├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization)
│ └── → PAIRWISE TOURNAMENT ✅ (compare each rollout vs all others in group,
│ reward = net win rate within group)
│
├── DPO / IPO / SLiC — need (chosen, rejected) pairs
│ └── → PAIRWISE ✅ (two-way comparison, return winner/loser)
│
└── Best-of-N / reranking — rank N candidates
└── → LISTWISE ✅ (single call ranks all N at once)
Cost constraint?
├── Low budget
│ └── FunctionGrader (free) → pointwise; or pairwise with small judge model
│
├── Medium budget
│ └── Pointwise: 2–3 LLM graders + WeightedSumAggregator
│ └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group
│
└── High quality / no cost limit
└── Pointwise voting (3–5 calls) or pairwise with strong judge + debiasing
Sub-documents — Read When Relevant
| Topic | File | Read when… |
|---|
| Pointwise multi-dim reward | pointwise.md | GRPO on verifiable tasks; multi-dimension scoring |
| Pairwise reward | pairwise.md | GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs |
Read the relevant sub-document before writing any code.
Install
pip install py-openjudge
Strategy Comparison
| Strategy | Output | Reward signal | Typical use | Cost |
|---|
| Pointwise | scalar per response | direct reward r(x, y) | GRPO on verifiable tasks, filtering | Low–Medium |
| Pairwise Tournament | net win rate per response | relative reward within group | GRPO on subjective tasks | Medium (N²/2 calls) |
| Pairwise | winner/loser pair | implicit preference y+ > y- | DPO, IPO, RLAIF preference data | Medium |
| Listwise | rank over N responses | ordinal reward / reranking | Best-of-N, reranking | Medium–High |
Score Normalization
All graders return scores on different scales. Always normalize before feeding into RL:
def normalize(score: float, min_score: float, max_score: float) -> float:
"""Map [min_score, max_score] → [0.0, 1.0]."""
if max_score == min_score:
return 0.0
return (score - min_score) / (max_score - min_score)
# LLM graders (common/*) return 1–5 → normalize to 0–1
reward = normalize(result.score, min_score=1, max_score=5)
# FunctionGrader / text graders already return 0–1 → no normalization needed
Evaluation Strategies
Evaluation strategies control how many times a grader is called and how
results are aggregated. They are independent of the grader itself.
Choose Your Strategy
Grader type?
│
├── Deterministic (FunctionGrader, StringMatch, CodeExecution, etc.)
│ └── → Direct (zero variance, no need for aggregation)
│
├── LLM grader — Pointwise scoring
│ │
│ ├── Budget limited / speed critical
│ │ └── → Direct (accept variance, 1× cost)
│ │
│ ├── Discrete scores (1–5 integer, pass/fail, binary)
│ │ └── → Voting (majority vote, robust to outliers, N× cost)
│ │
│ └── Continuous / fine-grained scores (need precise ranking)
│ └── → Average (mean, preserves signal, N× cost)
│
└── LLM grader — Pairwise GRPO tournament
└── → GRPOTournament (all-pairs comparison, net win rate)
| Strategy | Aggregation | Best for | Cost |
|---|
DirectEvaluationStrategy | None | Deterministic graders; low budget | 1× |
VotingEvaluationStrategy | Majority vote | Discrete / integer LLM scores | N× |
AverageEvaluationStrategy | Mean | Continuous LLM scores | N× |
GRPOTournamentEvaluationStrategy | Net win rate | Pairwise GRPO on subjective tasks | N²/2× |
All strategies are imported from openjudge.evaluation_strategy.
Pointwise — Noise Reduction with Voting / Average
For high-variance LLM judges, wrap any grader with VotingEvaluationStrategy
to run N calls and take the majority vote:
from openjudge.evaluation_strategy import VotingEvaluationStrategy
grader = CorrectnessGrader(
model=model,
strategy=VotingEvaluationStrategy(num_votes=3, tie_breaker="closest_to_mean"),
)
# Now each call internally runs 3 LLM evaluations and returns the most common score
Use odd num_votes (3, 5) to avoid ties.
Pairwise — GRPO Tournament
For GRPO on subjective tasks, use GRPOTournamentEvaluationStrategy to run
all-pairs comparison and compute net win rate per rollout:
from openjudge.evaluation_strategy import GRPOTournamentEvaluationStrategy
strategy = GRPOTournamentEvaluationStrategy(debiased=False)
results = await strategy.execute(
pairwise_grader.aevaluate,
query="Write a haiku about the ocean.",
responses=["rollout_1", "rollout_2", "rollout_3", "rollout_4"],
)
rewards = [r.score for r in results] # net win rates in [-1.0, 1.0]
Set debiased=True to run each pair in both orders and only count consistent
results (doubles LLM calls but mitigates position bias).