Run the LangSmith evaluation suite and display pass/fail results
The English summary is being prepared. Showing the original description.
See how to installReview the instructions the agent will receive before you install.
Ensure Docker is running
docker compose ps
If PostgreSQL is not up:
docker compose up -d
Run the evaluation suite
uv run dotenv -f .env run -- python scripts/run_eval.py
Note the experiment name printed (e.g. medox-<hash>).
Fetch and display results
Write the following script to /tmp/check_eval.py then run it:
from langsmith import Client
client = Client()
runs = list(client.list_runs(project_name='<experiment_name>', is_root=True))
print(f'Eval cases: {len(runs)}')
print()
passed, failed = 0, 0
for run in runs:
fb = list(client.list_feedback(run_ids=[str(run.id)]))
score = fb[0].score if fb else None
comment = fb[0].comment if fb else ''
prompt = (run.inputs or {}).get('prompt', '').strip()[:75]
status = 'PASS' if score == 1 else 'FAIL'
if score == 1:
passed += 1
else:
failed += 1
print(f'[{status}] {prompt}')
if comment and comment != 'OK':
print(f' -> {comment}')
print()
print(f'Result: {passed} passed, {failed} failed out of {len(runs)}')
Replace <experiment_name> with the value printed in step 2, then:
uv run dotenv -f .env run -- python3 /tmp/check_eval.py
Investigate failures
For any [FAIL], read the comment and:
src/medox/agent/scripts/run_eval.py/add-eval-case to add a regression case if a new edge case was foundReport summary
Print the final Result: N passed, M failed out of X line to the user.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Add a tool function to the Medox agent. Use when creating a tool file, registering it in the API loop, and writing its test.
The English summary is being prepared. Showing the original description.
Add a new E2E test case to tests/e2e/prompts.yaml for LangSmith evaluation. Use when adding interaction pairs to test, covering new ANSM drug classes, or rebalancing eval coverage.
The English summary is being prepared. Showing the original description.
Create a git commit with Medox conventions (co-author, conventional type/scope). Invoke manually with /commit — do NOT trigger automatically.
The English summary is being prepared. Showing the original description.
Audit code changes on the current branch and update docs/ + .claude/rules/ to stay in sync. Run before /pr.
The English summary is being prepared. Showing the original description.
Run all project linters (ruff, mypy, sqlfluff) and report results. Invoke with /lint.
The English summary is being prepared. Showing the original description.
Add a new Dagster asset to the pipeline. Use when creating a Bronze, Silver, or Gold layer asset.
The English summary is being prepared. Showing the original description.