Desktop GUI automation via OpenInterpreter — mouse, keyboard, screenshot, and OCR control for native macOS/Linux applications. Three modes: Library (Claude reasons, OI executes), OS subprocess (full autonomous computer use), and Local agent (Ollama, offline). This skill should be used when interacting with desktop apps that have no CLI or API, automating GUI workflows, reading screen content via OCR, or controlling mouse/keyboard.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
설치 방법 보기설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.
Desktop control for Claude Code via OpenInterpreter (62k stars, AGPL-3.0). Mouse, keyboard, screenshot, and OCR primitives backed by pyautogui + pytesseract.
| Mode | LLM | Script | Best For |
|---|---|---|---|
| Library | Claude Code (native) | Individual scripts below | Surgical GUI actions — Claude sees screenshots, reasons, dispatches actions |
| OS subprocess | Claude API (via OI) | oi_os_mode.py | Full autonomous computer use — delegate entire GUI tasks |
| Local agent | Ollama (offline) | oi_os_mode.py --local | Offline computer use, no API costs, privacy-sensitive tasks |
Use Library mode by default. Use OS subprocess to delegate self-contained GUI tasks. Use Local agent when offline or to avoid API costs.
Run once:
~/.claude/skills/open-interpreter/scripts/oi_install.sh
Installs open-interpreter[os] via uv, verifies pyautogui and tesseract, checks macOS permissions.
macOS permissions (one-time, manual):
Verify permissions:
python3 ~/.claude/skills/open-interpreter/scripts/oi_permission_check.py
The core pattern for GUI automation:
1. Take screenshot → oi_screenshot.py
2. Read PNG → Claude Read tool (native vision)
3. Decide action → Claude reasoning
4. Execute action → oi_click.py / oi_type.py
5. Verify → Take another screenshot
6. Loop until done
oi_screenshot.py — Capture screen, return file path with Retina metadata
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py --region 0,0,800,600
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py --active-window
Output (3 lines):
/tmp/oi_screenshot_1708789200.png
SCALE_FACTOR=2
SCREEN_SIZE=1512x982
oi_click.py — Mouse click by coordinates or OCR text
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 900 --y 600 --image-coords
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300 --double
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300 --right
--image-coords: auto-divides by Retina scale factor (use when coordinates come from screenshot image pixels)--text: OCR-based — screenshots, finds text via pytesseract, clicks center of matchoi_type.py — Keyboard input
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --text "hello world"
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --key enter
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --hotkey command space
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --text "search" --method typewrite
--method typewrite: character-by-character (use when clipboard is needed for other purposes)--hotkey: AppleScript on macOS for reliable modifier key handlingoi_find_text.py — OCR screen reading
python3 ~/.claude/skills/open-interpreter/scripts/oi_find_text.py --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_find_text.py --text "Price" --screenshot /tmp/ss.png
Returns JSON array: [{"text": "Submit", "x": 450, "y": 300, "w": 80, "h": 24, "confidence": 95}]
oi_computer.py — Unified dispatch for all actions
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py screenshot
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py click --x 450 --y 300
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py type --text "hello"
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py find --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py scroll --clicks 3
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py mouse-position
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py screen-size
macOS Retina displays render at 2x (or 3x) scaling. Screenshot image pixels differ from screen coordinates:
| Metric | Example (14" MBP) |
|---|---|
| Image pixels (screenshot) | 3024 x 1964 |
| Screen coordinates (pyautogui) | 1512 x 982 |
| Scale factor | 2x |
When estimating click targets from a screenshot image, use --image-coords on oi_click.py to auto-divide by the scale factor. The oi_screenshot.py output includes SCALE_FACTOR metadata.
For self-contained GUI tasks, delegate to OI's full agent loop:
python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py "Open Calculator and compute 2+2"
python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py --provider anthropic "Change the desktop wallpaper"
OI runs its own screenshot → analyze → act loop using the Claude API. Requires ANTHROPIC_API_KEY.
Run OI with a local vision model via Ollama:
python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py --local "What apps are open?"
Prerequisites:
ollama serveollama pull llama3.2-visionLimitation: Local models use OI's classic code-execution mode, not the screenshot-driven OS Mode (which requires Claude 3.5 Sonnet). Local mode generates and executes code to accomplish GUI tasks rather than using pixel-level screenshot analysis.
python3 scripts/oi_type.py --hotkey command space
sleep 0.5
python3 scripts/oi_type.py --text "Calculator"
sleep 0.3
python3 scripts/oi_type.py --key enter
python3 scripts/oi_screenshot.py > /tmp/ss_meta.txt
python3 scripts/oi_find_text.py --text "Total" --screenshot "$(head -1 /tmp/ss_meta.txt)"
python3 scripts/oi_click.py --text "Save"
python3 scripts/oi_click.py --text "Email"
python3 scripts/oi_type.py --text "user@example.com"
python3 scripts/oi_type.py --key tab
python3 scripts/oi_type.py --text "password123"
pyautogui.FailSafeException (enabled by default)[oi] click at (450, 300) button=left| Symptom | Fix |
|---|---|
oi_screenshot.py returns black image | Grant Screen Recording permission to terminal app |
oi_click.py / oi_type.py no effect | Grant Accessibility permission to terminal app |
| OCR finds no text | Verify tesseract: which tesseract && tesseract --version |
| Retina coordinates off by 2x | Use --image-coords flag on oi_click.py |
oi_find_text.py low confidence | Try larger text, ensure screen is not obstructed |
| OS Mode hangs | Verify ANTHROPIC_API_KEY is set, check OI stderr output |
| Local mode fails | Verify Ollama running (ollama list) and model pulled |
| File | Contents |
|---|---|
references/computer-api.md | OI Computer API reference — mouse, keyboard, display, clipboard |
references/os-mode.md | OS Mode usage, provider configuration, agent loop architecture |
references/safety-and-permissions.md | macOS permissions guide, safety model, failsafe configuration |
아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.
로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.
아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.
Search academic papers, build literature reviews, and synthesize research findings — combines Exa MCP (research_paper category, arxiv filtering) with arxiv-mcp-server for paper discovery, download, and deep analysis. Triggers on academic paper, literature review, research synthesis, arxiv, find papers, scholarly search.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
This skill should be used when creating, auditing, or maintaining AGENTS.md files and Codex CLI configuration for any project—including initializing AGENTS.md for cross-agent compatibility (Codex, Cursor, Copilot, Devin, Jules, Amp, Gemini CLI), generating config.toml or .rules files, scaffolding .agents/skills/, converting CLAUDE.md to AGENTS.md, or auditing existing agent configs for bloat and staleness. Complementary to codex-orchestrator (which executes subagents; this skill creates the config files they consume).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Academic research skill for Biblical Hebrew, Semitic linguistics, cuneiform studies, and comparative Ancient Near Eastern research. Provides Sefaria API for Hebrew Bible, CDLI/ORACC for cuneiform databases, and web discovery via Omnisearch, Exa, Firecrawl, and Obscura for finding scholarly sources across JSTOR, Perseus, Persée, Google Scholar, and academia.edu. Triggers on Hebrew quotes, cuneiform, Sefaria, ANE research, Minoan, search for scholarship, find papers, literature review, scholarly search, academic search, Genesis/Tehom, Ugaritic, Talmudic sources, extract from PDF, OCR academic.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Build comprehensive ARCHITECTURE.md files following matklad's canonical guidelines — bird's-eye views, ASCII/Mermaid diagrams, codemaps, invariants, and layer boundaries. Triggers on document the architecture, create ARCHITECTURE.md, map this codebase, architectural overview.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Generate physically-based atmospheric scattering shaders — sky domes, planetary atmospheres, LUT-optimized pipelines, depth-aware post-processing. Four modes — sky-dome, atmosphere-post, planet, lut. Triggers on atmospheric scattering, sky shader, sunset rendering, planet atmosphere, Rayleigh scattering, Mie scattering, volumetric sky, sky dome, atmosphere post-processing, aerial perspective, transmittance LUT, sky rendering, realistic sky, planetary rendering.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.