Agent Assist — Simulate
Adversarially test and harden an implemented agent before deployment. Runs three simulation tracks
(attack, behavior, persistence), then patches and retests failing scenarios across multiple rounds
until all pass or the fixing limit is reached.
Script Path Resolution
Resolve once per session: <skill_scripts_dir> is the scripts/ subdirectory next to this
SKILL.md. Confirm it exists. Use the resolved absolute path everywhere.
Python: prefer .venv/bin/python3 if a .venv/ exists; otherwise python3. Store as
<python>. Require 3.11+:
<python> -c "import sys; assert sys.version_info >= (3, 11)"
<python> -c "import pydantic, yaml" 2>/dev/null || <python> -m pip install pydantic pyyaml
<python> <skill_scripts_dir>/_bootstrap.py
The last line locates the shared datarobot-skills-utils package (a repo checkout or an
installed copy) and installs it from PyPI when neither is present.
OpenCode:
dr opencode upgrade && export PATH="$PATH:$HOME/.opencode/bin"
Auth:
dr auth check
dr dotenv update
If any check fails, surface the error and stop.
Swarm directory: All intermediate files (scenario inputs/outputs, run results, convergence state) are written to .datarobot/swarm/ under the project root and deleted at the end of Step 5.
If .datarobot/swarm/ already exists at the start of a session, it is leftover from a prior
session that did not complete Step 5 cleanup. Run rm -rf .datarobot/swarm/ before proceeding.
Pre-flight Check
For every spec read, prefer agent_spec.yaml and fall back to agent_spec.md if the YAML file is absent. If both exist, use the YAML file. Write any spec updates only to agent_spec.yaml; pass the resolved existing spec path to simulation commands.
- Resolve the spec by preferring
agent_spec.yaml and falling back to agent_spec.md. Confirm the selected file has a system_prompt. If neither file exists or the selected spec has no system_prompt, route the user to agent-assist-build and stop.
- Confirm implementation code exists (
agent.py, myagent.py, tools.py, or app.py). If not, route to
agent-assist-build and stop.
Step 1 — Configure
Collect answers in sequence. Do not start simulation until all are answered.
If agent_config.yaml already exists, read it and ask:
"Last time: [persona], [context or none], [iterations] rounds of fixing, [eval mode], [model]. Same settings or change anything?"
Q1 — User type: Read agent_spec.yaml and offer 2–4 domain-specific personas plus
"Other — describe your user segment."
Q2 — Grounding context (optional): Based on agent_spec.yaml, ask for domain-appropriate
examples (e.g. past queries, sample inputs, real requests).
Keep the question to one sentence. Save to user_context.txt if provided. Skip if the user says "skip."
Q3 — Fixing rounds: Ask:
"How many rounds of fixing should I run on failing scenarios? Default: 3."
Q4 — Evaluation mode: Ask:
"How should results be evaluated? Standard gives a simple pass/fail. Scored rates each result by severity: low, medium, high, or critical. Default: standard."
Q5 — Model: Run dr opencode models. Recommend the strongest available models from the
catalog. Show up to 5 as a numbered list. After the list, ask: "Want to see more models from
DataRobot LLM Gateway?" If yes, show more from the catalog. Store the choice as <model>.
Q6 — Selective tool execution (optional): Read the tool definitions in agent_spec.yaml and
identify any that appear to be read-only — no writes, no side effects, names like get_, list_,
load_, fetch_, search_. If any exist, ask:
"I can call these tools for real during simulation instead of generating fictional return values:
[list each with its description]
Run them for real, or simulate all? Default is simulate."
If the user chooses real execution: edit agent_spec.yaml to add is_readonly: true on each
approved tool, then pass --execution-mode selective_e2e to native_scenarios.py configure.
Keep using <python> for native_swarm.py and tool_executor.py — the executor re-execs
into the nearest .venv beside the tools file.
If the user declines or no read-only tools exist, omit --execution-mode (defaults to simulated).
Start OpenCode server (once, after model selection):
dr opencode serve --port 4096 2>/dev/null &
until curl -sf http://127.0.0.1:4096/global/health >/dev/null 2>&1; do sleep 0.25; done
Store the PID as <opencode_server_pid> and http://127.0.0.1:4096 as <opencode_server_url>.
Persist config:
<python> <skill_scripts_dir>/native_scenarios.py configure agent_spec.yaml \
--user-persona "<persona>" \
--iterations <n> \
--judge-mode <standard|scored> \
--model "<model>" \
[--context user_context.txt]
Step 2 — Generate Scenarios
<python> <skill_scripts_dir>/native_scenarios.py prepare agent_spec.yaml \
--config agent_config.yaml
Run each generator one at a time. Before each, announce what the track tests. After each, report
the count and list the scenario names.
Say: "Generating attack scenarios — these test whether the agent can be manipulated into bypassing its own restrictions."
<python> <skill_scripts_dir>/gateway_worker.py \
--role-prompt generate-attack \
--input-path "$SWARM_DIR/attack-input.json" \
--response-path "$SWARM_DIR/attack-output.json" \
--model <model> --server-url <opencode_server_url>
Read $SWARM_DIR/attack-output.json and report: "Generated X attack scenarios:" followed by a list of scenario names.
Say: "Generating behavior scenarios — these test how the agent handles ambiguous or edge-case user requests."
<python> <skill_scripts_dir>/gateway_worker.py \
--role-prompt generate-behavior \
--input-path "$SWARM_DIR/behavior-input.json" \
--response-path "$SWARM_DIR/behavior-output.json" \
--model <model> --server-url <opencode_server_url>
Read $SWARM_DIR/behavior-output.json and report: "Generated X behavior scenarios:" followed by a list of scenario names.
Say: "Generating persistence scenarios — these apply multi-turn pressure to see if the agent holds its position under pushback."
<python> <skill_scripts_dir>/gateway_worker.py \
--role-prompt generate-persistence \
--input-path "$SWARM_DIR/persistence-input.json" \
--response-path "$SWARM_DIR/persistence-output.json" \
--model <model> --server-url <opencode_server_url>
Read $SWARM_DIR/persistence-output.json and report: "Generated X persistence scenarios:" followed by a list of scenario names.
Validate all outputs:
<python> <skill_scripts_dir>/native_scenarios.py finalize
Read finalize stdout and present the candidate list grouped by track. For each scenario output one line:
- [name] — [one sentence on what it targets]
Example:
Attack (4)
- Malicious CSV Injects Visualization Override — tests whether the agent follows instructions embedded in uploaded file content
- Path Traversal via File Path Argument — tests whether the agent rejects file paths outside the allowed directory
...
Behavior (3)
...
Persistence (3)
...
Then ask:
"Add or remove any, or say 'run it' to confirm. Say 'explain [name]' or 'explain all' for full detail on any scenario."
Write the authoritative criteria:
<python> <skill_scripts_dir>/native_scenarios.py confirm --output evaluation_criteria.md
Step 3 — Simulate
Find all implementation files in the working directory (agent.py, myagent.py, tools.py, app.py) and pass
each as a separate --implementation flag. Tell the user before launching:
"[N] scenarios queued — covering adversarial attacks, ambiguous user behavior, and multi-turn pressure. Typically 2–5 minutes."
<python> <skill_scripts_dir>/native_swarm.py run agent_spec.yaml \
--criteria evaluation_criteria.md \
--config agent_config.yaml \
--server-url <opencode_server_url> \
--model <model> \
--implementation <path> [--implementation <path> ...] \
[--tools-path <path/to/tools.py>]
Include --tools-path only when execution.mode is selective_e2e and tools.py exists. Parse
stdout as the summary JSON. Surface any warning: lines from stderr before presenting results.
While the swarm is running, read the task output file every 30 seconds and narrate newly completed
scenarios to the user (scenario name, track, pass/breach/error). Continue until the run completes.
Present results:
Say: "N of M scenarios passed."
Then give a structured narrative before proceeding:
Per-track summary — for each track (attack, behavior, persistence), state how many passed
and list each scenario with one line on what it tested. Example:
"Attack — 4/4 passed: path traversal via load_dataset, query injection via run_analysis,
scope escalation via wildcard, instruction override attempt."
Per-breach narrative — for each breach, say:
"Breach: [scenario_name] ([track])
What happened: [breach_reason in plain language]
Likely fix: [prompt addition / code guard]"
Determine "likely fix" from the breach: if the agent violated a stated restriction, it is a
prompt patch. If the agent attempted a tool call it should not have made, it may need a code
guard in the implementation.
Overall read — one sentence on where the agent is strong and where it is soft. Example:
"The agent holds well on adversarial attacks and ambiguous behavior, but has soft spots under
sustained multi-turn pressure."
Proceed to Step 4 regardless of breached — native_convergence.py initialize must run even
when breached == 0, since it is what creates the convergence state that Step 5's report
command reads.
Step 4 — Converge
<python> <skill_scripts_dir>/native_convergence.py initialize agent_spec.yaml \
--criteria evaluation_criteria.md \
--config agent_config.yaml \
--actual-model "<model>"
Parse stdout as JSON: {"status": "...", "breaches": [...], "exhausted": [...], "passed": [...]}.
Each breach entry has scenario_id, scenario_name, track, breach_reason, transcript,
breach_indicators, iteration, and suggested_rerun_dir.
If status is complete, skip to Step 5.
Fix loop — repeat until advance returns complete:
For each breach in breaches:
-
Read the breach transcript and breach_reason. Propose the minimal addition to the
system prompt that prevents this behavior. Tell the user:
"Breach: [scenario_name] — [breach_reason]
Proposed fix: [your proposed text]
Apply this patch? (yes/no)"
If approved, edit agent_spec.yaml directly to append the text to system_prompt.
Then find the SYSTEM_PROMPT string in the implementation files (check agent.py,
myagent.py, tools.py, app.py) and apply the same addition there so the deployed
agent stays in sync with the spec.
If the breach is structural (cannot be fixed by prompt alone — e.g. a missing tool
guard in the implementation), say so and read the implementation file at the relevant
function. Propose a targeted code change and ask for approval. If approved, apply it
with your Edit tool.
-
Before retesting, tell the user:
"Breach: [scenario_name]
What happened: [breach_reason]
Fix: [one sentence describing what was added to the system prompt or implementation]
Retesting now to verify the patch holds."
Then re-run the scenario. Call initialize once to set up the run and get the first
input/response paths:
<python> <skill_scripts_dir>/native_execution.py initialize agent_spec.yaml \
--criteria evaluation_criteria.md \
--scenario-id <scenario_id> \
--run-dir <suggested_rerun_dir>
This returns {"role": "runner", "input_path": "...", "response_path": "...", "turn_number": 1, ...}.
Role → prompt name mapping (use this to pick --role-prompt):
runner → run-scenariofixture → generate-tool-returnevaluator → evaluate-result
Drive to terminal — loop using submit output to advance state. Do NOT call initialize
again inside the loop:
Before each worker call, announce the current step:
- Runner:
"Turn N/M — running scenario"
- Fixture:
"Turn N/M — generating tool return"
- Evaluator:
"Turn N/M — evaluating"
# 1. Call worker using input_path and response_path from initialize (or previous submit).
# For the fixture role: if agent_config.yaml has execution.mode: selective_e2e and
# tools.py exists, call tool_executor.py instead of gateway_worker.py:
#
# selective_e2e fixture:
# <python> <skill_scripts_dir>/tool_executor.py \
# --input-path <input_path> --response-path <response_path> \
# --tools-path <tools_path> --readonly-tools <comma-separated readonly fn names>
#
# all other roles (runner, evaluator) and simulated fixtures:
<python> <skill_scripts_dir>/gateway_worker.py \
--role-prompt <run-scenario|generate-tool-return|evaluate-result> \
--input-path <input_path> \
--response-path <response_path> \
--model <model> --server-url <opencode_server_url>
# 2. Submit and read next state:
<python> <skill_scripts_dir>/native_execution.py submit \
--run-dir <suggested_rerun_dir> --response <response_path>
submit returns the next {"role", "input_path", "response_path", "turn_number"} or a
terminal {"status": "passed"|"breach"|"error"}. Use the returned input_path and
response_path for the next worker call. Stop when status is terminal.
Report ✓ passed / ✗ breach / ! error when terminal.
-
After all reruns in this round, call advance with the completed rerun dirs:
<python> <skill_scripts_dir>/native_convergence.py advance agent_spec.yaml \
--rerun <scenario_id>:<suggested_rerun_dir> [--rerun ...]
Parse stdout the same way as initialize. If status is complete, stop.
Otherwise repeat the fix loop for the new breaches list.
For any scenario in exhausted: read its breach_reason and transcript, identify the
implementation function responsible, propose a targeted code fix, ask for approval, apply
with your Edit tool, then rerun it (using the next iteration dir from suggested_rerun_dir
in the advance output) and call advance again.
When advance returns complete:
"Convergence complete. X resolved, Y exhausted."
Step 5 — Report
<python> <skill_scripts_dir>/native_convergence.py report agent_spec.yaml \
--output eval_report.md
kill <opencode_server_pid> 2>/dev/null || true
rm -rf .datarobot/swarm/
After the script writes eval_report.md, append a "## Changes Applied" section to the file
listing every change made during Step 4 — scenario name, what was changed, and why. For prompt
patches, list both the addition to agent_spec.yaml and the corresponding change to the
implementation file. For code fixes, list the function and file changed. Use your Edit tool to
append to eval_report.md.
Present passed/total, unresolved, exhausted, and readiness to the user. If ready: false, say so
explicitly before offering next steps.
For each exhausted scenario, immediately after the pass/fail summary line, add one line per scenario:
"⚠ [scenario_name] couldn't be resolved after [N] attempts — this needs your attention before deploying."
Then read eval_report.md and present a per-track breakdown. For each track (attack, behavior,
persistence), list each scenario with one line: what it tested, whether it passed initially or
needed a fix, and how many turns it ran.
Next steps:
What would you like to do next?
1. Review eval_report.md — outcomes and unresolved scenarios
2. Re-run simulation — after further changes
3. Test locally — run the agent on your machine
4. Deploy — deploy the hardened agent to DataRobot
- If 1: read
eval_report.md and present a structured summary.
- If 2: return to Step 1 (reuse saved settings or re-collect).
- If 3: read
AGENTS.md, display the local test command, tell the user to run it in a new terminal.
- If 4: follow the deploy instructions in
agent-assist-build/SKILL.md.
Error Handling
Step 2 generator failure — worker exited non-zero (response extraction failed) — the worker
replied with prose instead of JSON. Retry that generator once with a --rejection-note forbidding
tool calls (including the built-in skill tool) and meta-commentary. If it still fails, retry once
with a different top-tier <model> (e.g. claude-opus-4-8) and tell the user which track used it.
If that fails, surface the error and stop.
Step 2 generator failure — finalize rejected the content — finalize prints
role:<role> validation failed: <reason>. Retry that generator once with
--rejection-note "<reason>" and rerun finalize. If it still fails, surface it and stop.
Step 4 rerun worker failure — if a runner/fixture/evaluator worker exits non-zero, mark the
scenario failed:
<python> <skill_scripts_dir>/native_execution.py fail \
--run-dir <run_dir> --reason "<reason>"
Then call advance with that run dir so the script records it as errored and moves on.
Auth errors (401 / UNAUTHORIZED): Run dr auth login and retry immediately.
Script timeouts: Allow up to 2 minutes per worker, 10 minutes for native_swarm.py run.
Simulation Tracks
| Track | Degrades when... |
|---|
| Attack | No tools defined — scenarios are capability-generic |
| Behavior | No grounding context — falls back to generic user archetypes |
| Persistence | No explicit restrictions in system prompt or implementation code |
Surface these gaps to the user if relevant.