本文へ移動
cccskills
無料GitHub で公開

swe-bench-grafema-experiments

SWE-bench pipeline for A/B testing Grafema with Claude Code. Use when: (1) running SWE-bench experiments, (2) comparing baseline vs grafema conditions, (3) evaluating results, (4) debugging container issues.

Disentinel/grafema
インストール方法を見る

含まれるファイル（1）

  • SKILL.md2.8 KB

SKILL.md（原文）

インストールする前に、エージェントに与えられる指示の中身を確認できます。

SWE-bench Grafema Experiments

Pipeline Overview

Uses claude -p inside SWE-bench Docker containers. Two conditions:

  • Baseline: Claude Code + standard tools
  • Grafema: Claude Code + grafema installed with pre-built graph + MCP tools

Same agent, same environment, same prompt (except tool docs section).

Quick Commands

# Generate tasks.json (one-time)
python scripts/swe-bench/generate-tasks.py > scripts/swe-bench/tasks.json

# Run single task
./scripts/swe-bench/run.sh axios__axios-4731 --mode baseline
./scripts/swe-bench/run.sh axios__axios-4731 --mode grafema

# Run all JS/TS tasks (loop)
for task in $(jq -r '.[].instance_id' scripts/swe-bench/tasks.json); do
  ./scripts/swe-bench/run.sh "$task" --mode baseline
  ./scripts/swe-bench/run.sh "$task" --mode grafema
done

# Compare results
./scripts/swe-bench/compare.sh

# Evaluate with swebench
source /Users/vadimr/swe-bench-research/mini-swe-agent/.venv/bin/activate
python -m swebench.harness.run_evaluation \
  --dataset_name swe-bench/SWE-Bench_Multilingual \
  --predictions_path scripts/swe-bench/results/baseline/preds.jsonl \
  --max_workers 1 --run_id baseline

Key Files

FilePurpose
scripts/swe-bench/run.shMain pipeline script
scripts/swe-bench/compare.shResults comparison
scripts/swe-bench/generate-tasks.pyTask generation from HuggingFace
scripts/swe-bench/tasks.jsonPre-cached 43 JS/TS tasks
scripts/swe-bench/templates/prompt-baseline.mdBaseline prompt
scripts/swe-bench/templates/prompt-grafema.mdGrafema prompt (with tool docs)
_ai/swe-bench-runbook.mdFull runbook

Debugging

Container issues

# Check image exists
docker images | grep sweb.eval

# Check Node version
docker run --rm <image> node --version

# Debug inside container
docker run -it --name debug-swe -v ~/.claude:/root/.claude:ro <image> bash

Auth issues

# Verify host auth
claude --version

# Check mount inside container
docker exec <container> ls -la /root/.claude/

Grafema issues

# Check if grafema installed
docker exec <container> grafema --version

# Check if graph was built
docker exec <container> ls /testbed/.grafema/

# Check MCP config
docker exec <container> cat /testbed/.mcp.json

Historical Results

Grafema consistently reduces file exploration (39-100%) but hasn't improved fix correctness in tested tasks. The new claude -p pipeline should provide cleaner measurements.

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

ログインしてレビューを書く

同じリポジトリのスキル

概要と使いどころ

beam-elixir-ast-gotchas

無料

Fix Elixir/Erlang AST processing bugs in Grafema beam-analyzer. Use when: (1) Elixir parser returns MODULE node but 0 functions/calls — body nesting issue, (2) Erlang parser crashes with "cannot convert list to string" on OTP 26+ — location format changed from integer to keyword list, (3) pipe operator |> creates spurious CALL nodes instead of desugared function calls — clause ordering bug, (4) multi-module .ex files return only the first module — missing __block__ handler, (5) installing Erlang/Elixir on macOS with outdated Xcode/Clang.

日本語の概要は準備中です。原文の説明を表示しています。

Disentinel/grafema362026年8月24日 更新

bugs-to-grafema-capabilities

無料

Fires after fixing any non-trivial bug or regression. Asks: could the graph have caught this as a guarantee? Pairs with reflection-in-and-on-action — picks up after "earliest catchable signal" and asks the next question: was that signal expressible in graph? Triggers: (1) after any non-trivial bug fix is verified working, (2) after a regression report (something used to work, broke), (3) during step 6 (knowledge extraction) of the workflow, (4) when reflection-on-action surfaces a "would have been catchable" signal. Outcome is a triage decision (graph-reachable? rule expressible? rule sound?) and either a draft Linear issue + guarantee proposal, or a recorded "graph capability gap" note. Never auto-creates guarantees.

日本語の概要は準備中です。原文の説明を表示しています。

Disentinel/grafema362026年8月24日 更新

code-server-playwright-automation

無料

Fix Playwright automation failures against code-server (VS Code in browser). Use when: (1) trust dialog blocks all clicks — "monaco-dialog-modal-block intercepts pointer events", (2) button:has-text() finds wrong buttons behind modal dialog, (3) keyboard shortcuts don't work — Meta vs Control inconsistency, (4) VS Code extension activity bar icon not found by aria-label, (5) panels show placeholder text despite extension being "Connected". Covers trust dialog dismissal, keyboard shortcut hybrid mode, extension panel selectors, and database connection verification for Grafema extension.

日本語の概要は準備中です。原文の説明を表示しています。

Disentinel/grafema362026年8月24日 更新

dataflow-gauntlet-methodology

無料

Systematic methodology for achieving 100% backward dataflow reachability in a new language. Create gauntlet fixture, write trace, diagnose gaps, fix analyzer/algorithm, iterate to 100%. Language-agnostic process. Use when: (1) adding a new language to Grafema, (2) auditing dataflow coverage for existing language, (3) user says "/dataflow-gauntlet".

日本語の概要は準備中です。原文の説明を表示しています。

Disentinel/grafema362026年8月24日 更新

docker-exec-background-process

無料

Fix docker exec hanging when starting background processes (servers, daemons) inside containers. Use when: (1) docker exec never returns despite using & or nohup, (2) background server started in container causes docker exec to hang indefinitely, (3) env_startup_command in SWE-bench or similar frameworks times out, (4) setsid/disown needed for proper process detachment in Docker. Root cause: docker exec tracks ALL processes in the exec session, not just the top-level PID.

日本語の概要は準備中です。原文の説明を表示しています。

Disentinel/grafema362026年8月24日 更新

docker-node-version-compat-modules

無料

Fix Node.js CLI tools crashing inside Docker containers when host-installed node_modules require a newer Node version than the container provides. Use when: (1) "SyntaxError: Invalid regular expression flags" with /v flag in string-width or similar packages, (2) node_modules installed on host with Node 20+ but container has Node 18, (3) `npm install` with file: protocol creates symlinks that break inside Docker, (4) pnpm workspace packages become broken symlinks in containers. Covers version detection, Node binary mounting, and npm install strategies for cross-version compatibility.

日本語の概要は準備中です。原文の説明を表示しています。

Disentinel/grafema362026年8月24日 更新

Disentinel のスキルをすべて見る

このスキルの問題を報告する