Scaffold a first eval suite for an agent: mine real failures into cases, write behavioural checks over traces, and generate the runner. Use when the user wants evals, regression tests for an agent, a golden set, or asks how to know a prompt/model change did not break things. Do NOT use for a single task's done-check (goal-test) or for auditing context (context-audit).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
설치 방법 보기설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.
Theory: Two kinds of checks and the full walkthrough in Evals practice. An eval suite is the same test after every change. Behavioural checks read the steps of a trace; end-to-end checks read only the result. Start behavioural: they are deterministic, run in seconds, and diagnose instead of just scoring.
traces/<id>.json —
a list of events including tool calls. If the user's harness is Claude
Code, the transcript already is the trace; wire up whatever copies or
converts it. No trace, no behavioural checks.cases.yaml. One entry per failure:- id: refund_1042
input: "Refund order #1042, customer says it arrived broken"
expect: looks up the order before replying; asks approval before refund
check:
- trace has get_order before send_reply
- trace has approval_request before refund
The expect line is for humans; the check lines are the test. Keep the
rule language tiny: trace has X, trace has X before Y, trace lacks X.
5. Generate check_traces.py. A small runner: load cases.yaml, parse
each rule with a regex, walk the tool-call list, print one line per case,
exit non-zero on any failure. Keep it dependency-light (pyyaml only) and
fast — the whole suite should run in seconds, with no model calls.
6. Run it and hand over the flywheel. Show the pass/fail lines. Then
leave the loop in writing at the end of your report: read fresh traces
weekly, add every new failure as a case, fix the biggest cluster, re-run.
python check_traces.py) so it can gate a
CI job or a pre-release habit without ceremony.아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.
로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.
아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.
Audit an agent's context layout against the four places: system prompt, tools, history, tail. Use when the user asks to audit, review or fix their agent's context, prompt caching, token spend per turn, or CLAUDE.md / AGENTS.md layout. Read-only: reports findings and a fix list, changes nothing. Do NOT use for writing evals (evals-bootstrap) or for loop design (goal-test).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Find the decisions in a pipeline that do not need the expensive model and propose the gate for each: a rule, a classic classifier, or a small model, with fail-closed routing. Use when the user asks to cut model costs or latency, says the big model handles everything, or wants a triage / routing / filter layer in front of an agent. Analysis plus an optional baseline scaffold. Do NOT use for auditing context layout (context-audit) or for building eval suites (evals-bootstrap).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Turn a vague task into a testable definition of done and generate an executable goal-test script for it, optionally with a bounded retry loop around a headless agent. Use when the user wants to run an agent in a loop, asks how to know when an agent task is finished, or says a goal like "improve X" needs to become checkable. Do NOT use for building eval suites over many cases — that is evals-bootstrap.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Audit an agent's harness against the four rings: containment, guides, sensors, permissions. Use when the user asks whether their agent setup is safe, wants a review of sandboxing / permissions / hooks / AGENTS.md guardrails, or asks what a bad session could break. Read-only: reports the blast radius and a hardening list, changes nothing. Do NOT use for context layout (context-audit) or cost routing (gate-check).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Import a large archive into the vault in controlled batches: triage what is worth ingesting, process oldest first, checkpoint after every batch, and keep cost visible. Use this skill whenever the user wants to import years of bookmarks, an export from another notes app, a downloads folder, a chat history dump, or says "backfill", "bulk import", "process my archive". Do NOT use for ingesting one or a few new sources, which is second-brain-ingest.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Trace where the user's recorded position on something has shifted: superseded claims, contradictions between sources of different dates, and what evidence moved them. Use this skill when the user asks what they have changed their mind about, how their thinking on a topic developed, what they used to believe, or wants a quarterly look back at their own positions. Do NOT use for finding contradictions between sources with no time dimension, or for a current-state answer.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.