Meta-Harness — proteus memory-summary evolution
Run ONE iteration. Do all work in the main session — do NOT delegate to subagents.
You do NOT run benchmarks. You analyze prior results, prototype a mechanism,
and write new candidate summary compressors. The outer loop (meta_harness.py)
scores them on (fidelity, chars) separately, with no model and no network.
What a candidate is
A summary compressor: it turns one campaign-memory record (a dict — see
corpus.py) into the short string injected into the policy's context on
retrieval. The proteus analog of a memory system. The grading is in
corpus.py::score_fidelity: the fraction of load-bearing facts (target,
surface, strategy, outcome, quality, difficulty, transfer hint) that survive in
your summary. Context cost = len(summary).
The objective
Preserve fidelity (>= the floor in config.yaml, currently 0.70 worst-record)
while using FEWER characters than agents/baseline_incumbent.py. The frontier
is Pareto: fidelity up, chars down. You cannot win by dropping facts — a summary
that loses a required fact loses fidelity and falls off the frontier.
CRITICAL CONSTRAINTS
- Implement exactly 3 new compressors this iteration.
- Each must change a mechanism, not a constant. Bad: "same template, drop the
organism." Good ideas: abbreviation/symbol encoding of fixed vocab
(surface types, outcomes); a key:value micro-syntax instead of prose;
dropping only provably-redundant words; reordering so the highest-value facts
survive truncation; field-name elision where the value is self-identifying.
- No record-specific hints. Never hardcode a target name, campaign_id, or
any value from
corpus.py into a compressor. It must generalize to unseen
records. (This is the anti-leakage rule — load-bearing for proteus.)
- Do not abort early or write "the frontier is optimal".
Workflow
- Analyze. Read
logs/evolution_summary.jsonl (what's been tried),
logs/frontier.json (current best), corpus.py (records + rubric),
agents/baseline_incumbent.py (the system to beat).
- Prototype (mandatory). Write a throwaway script in
/tmp/ that runs your
compression idea over a couple of corpus.py records and checks fidelity by
eye before committing. Delete it after.
- Implement. For each of 3 candidates: copy
agents/baseline_incumbent.py
to agents/<snake_name>.py, subclass SummaryCompressor, implement
summarize(self, record) -> str. Import from candidate_base. Self-critique:
is this a new mechanism or just a tweaked constant? If the latter, rewrite.
- Validate.
python -c "import agents.<name>; print('OK')" from the repo root.
- Write
logs/pending_eval.json:
{
"iteration": <N>,
"candidates": [
{"name": "<snake_name>", "hypothesis": "<falsifiable claim about fidelity/chars>"}
]
}
Output: CANDIDATES: <name1>, <name2>, <name3>
Interface
from candidate_base import Record, SummaryCompressor
class MyCompressor(SummaryCompressor):
def summarize(self, record: Record) -> str:
... # pure, deterministic, no I/O, no LLM