Peekaboo

Use native Peekaboo commands for macOS apps, windows, menus, dialogs, browser chrome, and pixels. Prefer existing browser tooling for page content, DOM, forms, console, and network; Peekaboo's browser command is available when that integration is configured. Stay within the user's authorized task and verify the resulting UI.

For Peekaboo browser work, retain one Bridge host and its existing provider connection across calls. The pinned Chrome DevTools MCP 1.10.1 is launched through Peekaboo's audited telemetry bootstrap; a bare upstream npx invocation does not carry that fix. Background calls require an existing exact connection and retain the source-audited tool allowlist. Page evaluation can grant web user activation even when Chrome remains behind another app, so do not infer background safety from window focus alone or retry a refused call with foreground authority implicitly.

Select the CLI and execution host

Use the installed signed CLI for ordinary automation; building Peekaboo is a development task, not a prerequisite. Honor an explicit binary override and inspect its version:

PB="${PEEKABOO_BIN:-$(command -v peekaboo)}"
"$PB" --version
"$PB" bridge status --verbose --json
"$PB" permissions status --all-sources --json

The CLI ships separately from Peekaboo.app. Keep the release archive's CLI and accompanying compatibility dylibs together. The app's executable is a GUI host, not the CLI; current releases reject CLI-style arguments before Bridge startup. Using the same current signed CLI/app release avoids capability drift, but compatibility is negotiated by protocol and operation capabilities, not simple version equality.

Host selection depends on the operation. Ordinary automation prefers a healthy reusable daemon, then a capable GUI host, before starting a daemon. Capture, AX inspection, browser, and snapshot-state work first prefer and may start the current CLI build's daemon. An explicit snapshot routes to its unique live producer; explicit socket or local-only overrides remain authoritative. Do not stop hosts merely because several candidates exist.

When app-held permissions are needed, launch the GUI host and pin the same socket for permission checks and observation:

open -gj -a Peekaboo
GUI_SOCKET="$HOME/Library/Application Support/Peekaboo/bridge.sock"
"$PB" bridge status --bridge-socket "$GUI_SOCKET" --verbose --json
"$PB" permissions status --bridge-socket "$GUI_SOCKET" --json
"$PB" see --bridge-socket "$GUI_SOCKET" --no-elements --mode screen --path /tmp/peekaboo-screen.png --json

open -gj requests a background/hidden launch; first-run onboarding or missing-configuration prompts can still activate the app. Verify the reported host rather than assuming it was selected. The GUI app, daemon, and caller-local CLI have separate TCC contexts. Screen Recording is checked where capture runs, Accessibility where AX runs, and Event Synthesizing where events are sent. Both local and Bridge sources need not have identical grants.

Observe, target, act, verify

1. Resolve the target with app list and window list; prefer exact PID/window IDs over broad app names or titles.
2. Observe with pixels, Accessibility, or both. Ordinary see does not activate the target; --web-focus explicitly permits a focus action and is not a read-only retry.
3. Copy opaque element and snapshot IDs from that observation. Prefer an element click, set-value for an intended value replacement, or a suitable state-only AX action over coordinate input.
4. Use background delivery with an exact target. Add --foreground when shared-desktop interaction is authorized and needed; do not silently promote a refused background command.
5. Read the canonical outcome and verify the intended change with a fresh see, verify, or other readback. A successful dispatch alone does not prove the app changed.

"$PB" app list --include-hidden --include-background --json
"$PB" window list --app Safari --json

# Read clipboard contents only when relevant to the authorized task.
"$PB" clipboard get --json

# Pixels only; an exact window also publishes a coordinate receipt.
"$PB" see --window-id "$WINDOW_ID" --no-elements --path /tmp/peekaboo-window.png --json

# Pixels and element IDs, with an annotated artifact.
"$PB" see --window-id "$WINDOW_ID" --annotate --path /tmp/peekaboo-elements.png --json

# AX text and IDs without screenshot capture.
"$PB" see --window-id "$WINDOW_ID" --tree --no-screenshot --json

# Use IDs from the applicable fresh observation, not all examples in sequence.
"$PB" click --on "$ELEMENT_ID" --snapshot "$SNAPSHOT_ID" --json
"$PB" scroll --direction down --on "$ELEMENT_ID" --snapshot "$SNAPSHOT_ID" --json

After a mutation changes the UI, capture again before using IDs for another action. requires_fresh_observation: true consumes the snapshot for mutation. Partial, indeterminate, and unverified results may already have changed the app; never repeat input blindly. Inspect image contents to verify capture and visual results. sips -g pixelWidth -g pixelHeight <path> checks dimensions only.

Background input and coordinates

Background click can invoke an element's AX action or focus a writable text field without activating its app. --input-strategy actionOnly selects the AX route; coordinate clicks can also use it by hit-testing a pressable AX element. Generic action AXPress and action AXShowMenu require --foreground; only generic AXIncrement/AXDecrement are background-safe. Prefer dedicated click for background activation.

Background typing/paste accept exact window routes even when the app has several windows. App/PID-only routes require a complete inventory with at most one eligible window. Prefer a fresh exact-window snapshot for type; it must identify the focused field in the intended internal key window. If needed, click the field, observe again, then type using the new snapshot. Exact routes revalidate process generation, window bounds, and focus. AX text edits need Accessibility; event fallback needs Event Synthesizing.

"$PB" type "text" --snapshot "$SNAPSHOT_ID" --json
"$PB" press Return --app TextEdit --window-id "$WINDOW_ID" --json
"$PB" paste "text" --app TextEdit --window-id "$WINDOW_ID" --json

type normally succeeds only for a confirmed change. It can return non-success after accepted dispatch, so read the outcome before retrying. Scripts doing their own follow-up observation may opt into --accept-dispatched; this does not confirm delivery. Raw background press needs an exact window selector or fresh exact-window snapshot; app/PID-only chords require --foreground. Default background-only Agent/MCP typing is stricter than the direct CLI: it requires a fresh exact non-dialog snapshot without competing selectors.

Background scroll accepts an element --on or coordinates --at x,y; app/window flags alone are insufficient. Coordinates require an explicit fresh screenshot --snapshot (never latest) and a protocol-1.43 host advertising backgroundCoordinateScroll. CLI points are window-local logical points; --global changes the basis without granting foreground authority. MCP coords defaults to global display points; image/normalized points use the observation's explicit coordinate_reference. Older hosts refuse without fallback.

Coordinate scrolling prefers the nearest proven AX scroll owner and can use exact-window wheel delivery on eligible native-WebKit surfaces without an exposed owner. Preserve unverified/retry-unsafe outcomes and verify the intended region actually moved. Targetless scrolling, --smooth, and nonzero --delay need --foreground and cannot be combined with --at. Shared-cursor move, targetless keyboard input, and click --long-press also require explicit foreground mode.

Background drag requires an explicit fresh --snapshot, with both endpoints inside its exact window. Use global logical coordinates or element IDs from that snapshot, a linear path, 1–10000 ms duration, and 1–96 samples. Protocol 1.39 hosts support this window-routed gesture without moving the physical cursor or activating the app; older hosts refuse without foreground fallback. Dispatch remains unverified and retry-unsafe, including an interrupted gesture: observe before another input. Cross-window/app drops, modifiers, human profiles, and app/window selector flags require explicit --foreground.

click --at uses logical points. Background coordinates are relative to the snapshot window even without target flags; add --global for global logical coordinates inside that window. Foreground coordinates are window-relative with target flags and global without them. Ordinary see produces logical1x images; do not divide their coordinates by a Retina scale. For --retina, crops, or resized previews, use the capture's coordinate metadata and screen list bounds/scale to map pixels to logical points.

"$PB" see --window-id "$WINDOW_ID" --no-elements --path /tmp/peekaboo-click.png --json
# Copy the returned snapshot ID before clicking window-local logical points.
"$PB" click --window-id "$WINDOW_ID" --at 20,40 --snapshot "$SNAPSHOT_ID" --json

Background coordinate clicks require a fresh screenshot snapshot with an exact process/window/bounds receipt; AX-only observations are insufficient. Moved, replaced, or unverifiable targets fail before dispatch. Background right/double/middle/triple clicks can use exact routed events, but completed dispatch remains effect-unverifiable. Observe before deciding whether another click is needed.

Troubleshooting and references

• Capture permissions belong to the executing host. Compare permissions status --all-sources, then pin the intended host for diagnostics and capture. Request only permissions needed for the selected operation; do not replace signed installs with ad-hoc builds against saved TCC/Keychain state.
• ScreenCaptureKit coordination is scoped to Peekaboo's own CLI/app processes and selected Bridge hosts. Other apps such as Claude, OpenClaw, Zoom, or OBS may run concurrently without blocking capture. Explicit see --capture-engine classic (alias cg) avoids in-process ScreenCaptureKit on the selected host; all engines still require valid capture permission evidence there.
• Prefer Bridge capture from SSH or background launchd sessions. --no-remote --capture-engine cg is a caller-local diagnostic for a known active Aqua session; elsewhere it can return wallpaper-only or redacted pixels despite reporting success.
• If a concrete snapshot's producer is unavailable or an explicit host does not own it, observe again on the intended host. Do not substitute an unrelated snapshot or weaken the target merely to retry.
• Use capture live for change-aware capture and capture video to sample an existing video. Store task captures under an explicit temporary path and inspect/redact them before any authorized sharing.
• Discover current syntax with <command> --help, learn, tools --json, and tools describe <name> --json (MCP schema, not CLI flags). Timing options accept bare milliseconds or ms/s; prefer explicit suffixes.
• v4 uses noun inventories (app list, window list, screen list), see, press, action, and click --at; use live help when translating older examples.

Canonical references stay usable when this skill is linked into another repository or agent directory:

• Command index, especially type, click, and scroll.
• Bridge host, permissions, and subprocess integration.
• For Peekaboo development, read the checkout's AGENTS.md and building guide; follow their source-build and test workflow for code changes.