Computer Use & OS Control MCP — Integration and AI Agent Guide

This skill file instructs AI agents on how to control macOS, Windows, or Linux desktop environments using the dual-platform computer-use-mcp servers. It details tool hierarchy, accessibility automation, scripting, window focus strategies, and troubleshooting steps.

1. Overview and Pipeline Value

The OS Control MCP Servers act as the ultimate fallback when structured application APIs do not exist. Agents use them to configure system settings, install dependencies, bypass modal dialogs, crop regions, perform OCR, and automate graphical interfaces.

Dual-Engine Architecture

• macOS & Linux (zavora_computer_use): @zavora-ai/computer-use-mcp@7.4.0, started via npx -y (pinned, nothing bundled). A TypeScript server plus a Rust native module for OS calls; also supports Windows. Requires Node.js 20+.
• Windows (bdb_windows_computer_use): Powered by a Python/PyWin32 automation bridge using pywinauto, pyautogui, comtypes, and local pytesseract OCR for deep Windows Desktop UI discovery.

---

2. System Instructions

Workflow Priorities

1. Discovery first: Call get_tool_guide if the server offers it (macOS; unverified for 7.4.0) or query window titles/automation trees (Windows) to evaluate if target apps support scripts/PowerShell or accessibility trees.
2. Accessibility over Coordinates: Use native Win32 Automation IDs / macOS Accessibility labels (AXButton, AXTextField, etc.) instead of mapping pixel coordinates from a screenshot. It survives window resizes, DPI changes, and layout changes.
3. Window Focus Strategies:
   • Verify focus before writing key streams.
   • Use strict focus for keyboard inputs to force the server to verify the target window is frontmost and visible before typing, preventing key drops.
   • Use prepare_display to automatically hide background apps that might steal focus or pop up notifications during execution.

---

3. Available Tools and API Parameters

macOS & Linux Engine (zavora_computer_use)

Default profile exposes 70 tools (COMPUTER_USE_PROFILE=core|ax|scripting|windows-admin|full changes the set). Ask the agent to run doctor to check capabilities and discover_applications to find apps. The tool names below are from the 6.2 era; the 7.4.0 npm README does not list tool names, so they are UNVERIFIED for 7.4.0. Check the server's tools/list before relying on them. Key tools (6.2):

• run_script(language, script): Runs AppleScript, JXA, or Shell commands.
• get_ui_tree(target_app, target_window_id): Returns a JSON structure of active UI roles and labels.
• click_element(role, label): Performs a semantic click on a button or menu.
• set_value(role, label, value): Sets values in text fields directly.
• screenshot(width, quality, target_window_id): Captures the viewport.
• left_click(coordinate, target_app): Simulates a left-click.

Windows Automation Engine (bdb_windows_computer_use)

Exposes 22 specialized tools. Key tools include:

• mouse_move(x, y) / left_click(): Simulates physical mouse interactions.
• type(text, press_enter): Types strings utilizing SendInput.
• screenshot(): Grabs active monitor outputs.
• get_ui_tree(backend): Dumps the complete Windows UIAutomation or MSActiveAccessibility control tree.
• ocr(region): Executes local Tesseract OCR on a screen subregion to extract text without cloud APIs.
• find_element(criteria): Resolves a control using criteria (title, class, automation_id).

---

4. Code Recipes and Prompt Cookbook

Recipe 1: Retrieve and Click elements on Windows via UI Automation

Instead of coordinates, locate the Notepad "File" menu item on Windows semantically:

// Step 1: Find target window and element info
// Tool: find_element(criteria={"title": "Notepad", "control_type": "Window"})

// Step 2: Semantic click on File menu button
// Tool: click_element(role="MenuItem", label="File")

Recipe 2: Run JXA (Javascript for Automation) Script on macOS

Create a new note in Apple Notes and write text programmatically:

// Tool: run_script(language="javascript", script="...")
// Script content:
var Notes = Application("Notes");
var newNote = Notes.Note({
    body: "<h1>BDB OS Integration Checklist</h1><p>Automated desktop test complete.</p>"
});
Notes.folders.byName("Notes").notes.push(newNote);
Notes.activate();

---

5. Troubleshooting and Connection Details

Configuration and Setup

• Windows Pre-requisites:
  • Python >=3.12 is required. The installer auto-compiles dependencies with uv.
  • For full OCR support, ensure Tesseract OCR is installed on Windows (winget install UB-Mannheim.TesseractOCR).
• macOS Permissions Required: You must grant Accessibility and Screen Recording access to your editor (e.g. Cursor, Claude Desktop, or Terminal) in System Settings -> Privacy & Security.

Common Errors and Fixes

1. FocusFailure: Target window is not frontmost
   • Cause: The strict focus strategy detected another application stole active focus.
   • Fix: Switch your focus_strategy to prepare_display to auto-hide background applications, or call activate_window first.
2. pywinauto.findwindows.ElementNotFoundError
   • Cause: The Win32 backend failed to resolve the title or control.
   • Fix: Switch backends (e.g., from win32 to uia) or verify if the app requires Admin escalation.