Ultimate Browsing

Web access for the hosts the user's browser cannot open (WAF / 403 / Cloudflare / bot-scored pages) and for platforms with a native reader. Reading and interaction on a page the user's browser can open belong to the browser skill's attached engine, not here. Start at the cheapest tier that can reach the blocked host and climb only when it cannot:

Tier 1 — insane-search (headless extraction + WAF bypass) -> Tier 1.5 — agent-reach (platform-native APIs, esp. Chinese platforms) -> Tier 2 — a real browser through omowright from js eval: 2a the owned engine (a browser your code launches, CloakBrowser for stealth), 2b the attached engine (the user's own signed-in browser).

PHASE 0 — ROUTE FIRST (MANDATORY)

User request
  |
  +- a page the user's browser can open (read, click, screenshot, QA, login)  NOT this skill (browser skill, attached engine)
  +- simple search query ------------------------------ NOT this skill (use web-search)
  |
  +- URL blocked / 403 / Cloudflare / WAF ------------- TIER 1  insane-search
  +- YouTube/Vimeo/TikTok subtitles or metadata ------- TIER 1  insane-search (yt-dlp)
  +- archive snapshot of a dead or changed page ------- TIER 1  insane-search (surrogates, provenance-tagged)
  |
  +- Chinese platform (xhs/douyin/weibo/bilibili/v2ex/wechat)  TIER 1.5 agent-reach
  +- podcast transcript / stock forum ----------------- TIER 1.5 agent-reach
  +- Twitter feed / LinkedIn profile / GitHub via CLI - TIER 1.5 agent-reach
  |
  +- Tier 1/1.5 returned empty or partial ------------- TIER 2  2a owned engine (stealth)
  +- a blocked host that also needs a click, a scroll or a screenshot  TIER 2  2a owned engine
  +- the user's account on a host the owned engine cannot pass  TIER 2  2b attached engine (their browser)

Read the matching reference before acting: references/insane-search/README.md, references/agent-reach/README.md, or references/chrome-stealth.md.

Tier 1 — insane-search (headless extraction)

When: blocked-URL bypass, media metadata, archive surrogates — no browser UI needed. Why first: ~10x faster than a browser, no process spin-up; handles most "fetch this blocked page" requests via curl_cffi TLS impersonation, yt-dlp (1858 sites), official public APIs, mobile URL transforms, Phase-2.5 surrogate archives (Wayback / archive.today snapshots, provenance-tagged — see references/insane-search/cache-archive.md), a key-gated Jina Reader (JINA_API_KEY), and a Playwright real-Chrome fallback. The engine lives inside this skill at engine/ and is invoked as a module. Surrogate results are dated COPIES: a result whose provenance is snapshot must be reported with its snapshot_timestamp, never presented as the live page.

# Core command — auto-detects WAF, runs the full fetch grid (run from the skill dir):
python3 -m engine "https://example.com/blocked-page"
#   add --selector "<CSS>" for positive-proof validation, --device auto|desktop|mobile,
#   --trace to inspect every attempt, --json for machine-readable output.

# YouTube subtitles / metadata (no browser):
yt-dlp --write-sub --write-auto-sub --sub-lang "en,ko" --skip-download -o "/tmp/%(id)s" "<URL>"

# Reddit / HN / Bluesky / arXiv etc. use official public endpoints — see the Phase 0 index in
# references/insane-search/README.md (Twitter syndication, Reddit .json, HN Firebase, ...).

The full engine harness (rules R1-R7, the Phase 0 official-API index, the no-site-name rule, and the references/insane-search/*.md deep-dives for TLS, Playwright routing, Naver, media, etc.) is in references/insane-search/README.md. Read it before tuning the engine or adding a WAF profile.

Escalate to Tier 1.5 or Tier 2 when

• The target is a Chinese / social platform with a native reader -> Tier 1.5.
• insane-search returns empty/partial, or the blocked page needs JS interaction, a screenshot, or media playback -> Tier 2.

Tier 1.5 — agent-reach (platform-native readers)

When: the target is a platform with a first-class API/CLI that beats generic fetching — especially Chinese platforms that stealth browsers still cannot reach cleanly. Several channels are zero-config (Douyin, V2EX, Reddit, RSS, YouTube); others need a one-time auth you supply via environment variables if you have access (JINA_API_KEY for Jina Reader — anonymous access is dead, see references/insane-search/jina.md; TWITTER_* for X; a transcription key for podcasts).

Category	Platforms	Entry
social	xhs (Xiaohongshu), douyin, weibo, bilibili, V2EX, Reddit, Twitter/X	references/agent-reach/social.md
web	Jina Reader, WeChat articles, RSS	references/agent-reach/web.md
video	YouTube, Bilibili, podcast transcripts, Douyin video	references/agent-reach/video.md
career	LinkedIn	references/agent-reach/career.md
dev	GitHub (gh CLI)	references/agent-reach/dev.md
search	Exa AI	references/agent-reach/search.md

mcporter call 'douyin.parse_douyin_video_info(url: "<URL>")'   # douyin, zero-config
curl -s "https://r.jina.ai/https://weibo.com/<uid>/<pid>"      # weibo via Jina
yt-dlp --dump-json "<bilibili-url>"                            # Bilibili (overseas: add --cookies-from-browser)
curl -s "https://www.v2ex.com/api/topics/hot.json"            # V2EX public API

Routing table, per-platform auth (set TWITTER_* env vars, gh auth login, a transcription key — only if you have access), rate-limit notes, and known version quirks are in references/agent-reach/README.md.

Tier 2 — a real browser (real interaction)

When: Tier 1/1.5 failed on a blocked host, or that host needs interaction (clicks, scroll, screenshots, video). A page the user's browser can open is the browser skill's job, attached engine first.

Both tiers are omowright, staged inside the browser skill and loaded from js eval:

const { loadOmowright } = await import("<browser-skill-root>/scripts/omowright.mjs")
const { omowright } = await loadOmowright()

Tier 2a — owned engine (default)

A browser your code launches with a task-owned profile. connectPipe opens no listening port; connectCloakProfile launches CloakBrowser with a pinned fingerprint seed and is the path for WAF, Cloudflare and bot-scored pages.

const browser = await omowright.connectPipe({ browserPath, browserArgs: ["--headless", `--user-data-dir=${profile}`], storageRoot: profile })
try {
  const page = await browser.newTab(url)
  const tree = omowright.compactSnapshot(await page.snapshot())   // the read; refs come from it
  const snoop = omowright.createNetworkSnoop(page)                  // read the API JSON instead of the DOM when there is one
  await page.locator("e3").click()
  await Bun.write(pngPath, await page.screenshot())
} finally {
  await browser.close()                                            // then rm -rf the profile
}

The rest of the surface (CUA coordinates, captcha solving, routes, traces, frames, human handoff) is in the browser skill's references/owned-engine/. A stealth binary is not proof of access: inspect the rendered result and report challenges that remain.

Tier 2b — attached engine (the user's account on a blocked host)

When the blocked flow needs the user's account, drive the browser they are already signed into instead of cloning their profile: connectBrowserSkill() → session.navigate → bskSnapshot(session) / session.observe() → session.click → session.stop(). NEVER launch against or clear cookies/cache/site data from the user's live profile, and never fall back to the owned engine for an authenticated criterion: if no extension is connected, run the browser skill's onboarding script and relay its one human step. The full loop is the browser skill.

Cookie login (cross-platform)

scripts/extract_cookies.py reads cookies from a local Chromium-family or Firefox-family browser and optionally injects them into the running CDP session. It resolves browser profile paths and decrypts cookie values per-OS (macOS Keychain, Linux libsecret, Windows DPAPI):

# Extract cookies to a file:
mkdir -p ~/.local/state/omo-cookies
python3 scripts/extract_cookies.py --browser chrome --domain youtube.com --output ~/.local/state/omo-cookies/youtube.cookies.json
# Extract and inject into the running CDP session:
python3 scripts/extract_cookies.py --browser chrome --domain youtube.com --inject --cdp 9242

Cookie export files are written with owner-only 0600 permissions. Do not place live auth cookies in shared temp directories or commit them to a repo. Cookie injection sends values to CDP over stdin rather than argv. Cookies apply on next navigation — reload after injecting. Google services use fingerprint-bound tokens that may not transfer across browser profiles. Limits in references/chrome-stealth.md.

Reference docs

File	When to read
references/insane-search/README.md	Tier-1 engine harness (R1-R7, Phase 0 API index, no-site-name rule) + its *.md deep-dives
references/agent-reach/README.md	Tier-1.5 routing table, platform auth, per-category *.md
references/chrome-stealth.md	Tier-2 stealth through omowright + CloakBrowser, cookie login limits

Environment variables

# agent-reach auth: set the channel-specific env vars from each tool's docs only if you have access
# insane-search needs no env vars — it auto-installs deps on first run

Anti-patterns

• Do NOT launch Chrome stealth for plain text extraction — use Tier 1.
• Use stealth plugins only in an explicitly installed script environment, not injected into WebView.
• Close every WebView/browser context when done and remove only task-owned profile clones.
• Do NOT inject cookies without reloading the page.
• Do NOT hardcode site domains/selectors into engine/** or waf_profiles.yaml — runtime hints only (see the no-site-name rule in the insane-search reference).