SkillOpt-Sleep: usage-driven self-evolution for the dsh agent

SkillOpt-Sleep is Microsoft's SkillOpt deployment-time companion engine: it reviews your past sessions (harvest), mines recurring tasks (mine), replays them through a selected backend (replay), and consolidates what it learns into skill documents behind a held-out validation gate (consolidate).

This skill drives the engine through the 7 skillopt_* tools exposed by the dsh-skillopt plugin. The default mock backend makes no model calls, which is useful for verifying the plumbing; a real backend consumes your API budget.

When to use

• "make my agent better the more I use it" / "learn my preferences across sessions"
• a one-off offline self-evolution / sleep / dream run (immediate or scheduled)
• review past sessions/trajectories and distill recurring tasks
• consolidate feedback into AGENTS.md / SKILL.md / managed skills
• schedule (cron) the cycle, or adopt a staged proposal

The cycle (six stages)

1. Harvest — read-only scan of supported local session records → digests
2. Mine — digests → recurring task records (intent + outcome labels + checkable refs)
3. Replay — re-run tasks under the current skill+memory with the selected backend → (hard, soft) scores
4. Consolidate — reflect on failures → propose bounded edits → validation gate on a held-out slice (default: accept only on strict improvement)
5. Stage — write accepted proposals to <project>/.skillopt-sleep/staging/<timestamp>/. Live files are unchanged. A rejected run still has a report but no proposal files.
6. Adopt — explicit (or operator-configured --auto-adopt) copies staged files over live ones, backing up first.

Driving it

Prefer the tools over hand-editing files:

Tool	Behavior
skillopt_status	state, engine availability, latest staged proposal & report
skillopt_dry_run	full preview (harvest+mine+replay), stages nothing
skillopt_run	full cycle, stages a proposal (live files unchanged by default)
skillopt_adopt	apply latest staged proposal (with backup) — the live-change boundary
skillopt_harvest	read-only show/export of mined tasks
skillopt_schedule / skillopt_unschedule	install/remove the nightly cron entry for this project

Typical flow:

# 1. check state (default mock backend, zero cost)
skillopt_status

# 2. preview the cycle
skillopt_dry_run project=<dir> source=<claude|codex|…>

# 3. real run (consumes the selected backend's API budget)
skillopt_run project=<dir> backend=<codex|claude|…> preferences="Prefer pytest; keep commits imperative."

# 4. review the report, then adopt
skillopt_adopt project=<dir>

# 5. schedule nightly at 03:17
skillopt_schedule project=<dir> hour=3 minute=17 backend=<codex>

Parameters

Parameter	Default	Meaning
project	config or cwd	project directory to evolve
backend	mock	mock|claude|codex|copilot|cursor|pi|opencode|handoff|azure_openai (mock = no model calls)
source	config	transcript source: claude|codex|copilot|cursor|pi|opencode|auto
model	backend default	replay model override
maxTasks	40	mined-task cap
preferences	empty	house rules for the reflection prior (e.g. "always use async/await")

Configuration (cordis.yml / bundle patch)

- insert:
    - id: skillopt
      name: './src/index.js'
      config:
        backend: codex
        project: /path/to/project
        preferences: 'Always use async/await'
        # auto-adopt is OPERATOR-ONLY — the model cannot set it
        autoAdopt: false

Advanced engine keys go in ~/.skillopt-sleep/config.json: gate_mode (on/off), gate_metric (hard/soft/mixed), gate_no_regression, dream_rollouts, recall_k, evolve_memory / evolve_skill.

Hard rules

• Never hand-edit AGENTS.md / SKILL.md around skillopt_adopt; let the engine's explicit adopt (or operator-configured --auto-adopt) apply the staging manifest, backing up live files first.
• Harvest is read-only; mock replay has no side effects.
• Real backends send truncated transcript excerpts and derived tasks to the selected provider for mining/replay/judging/reflection. For sensitive sessions, export tasks first (skillopt_harvest output=<file>), redact, set the top-level "reviewed" to true, then replay with --tasks-file; real backends refuse unreviewed task files.
• Show the user the held-out baseline → candidate score and the exact proposed edits before suggesting adoption. Evidence before adoption.

Validate / demo (no API spend)

pip install skillopt
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves

Deterministic synthetic demo: the score rises and the gate blocks a regression. It validates the mechanism, not effectiveness on your own tasks.

See the SkillOpt-Sleep docs for recorded results and limitations.