When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
日本語の概要は準備中です。原文の説明を表示しています。
実行スコア 未検証
インストール方法を見るインストールする前に、エージェントに与えられる指示の中身を確認できます。
Guides configuration and auditing of robots.txt for search engine and AI crawler control.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing goals.
Identify:
https://example.com)| Point | Note |
|---|---|
| Purpose | Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet) |
| Advisory | Rules are advisory; malicious crawlers may ignore |
| Public | robots.txt is publicly readable; use noindex or auth for sensitive content. See indexing |
| Tool | Controls | Prevents indexing? |
|---|---|---|
| robots.txt | Crawl (path-level) | No—blocked URLs may still appear in SERP |
| noindex (meta / X-Robots-Tag) | Index (page-level) | Yes. See indexing |
| nofollow | Link equity only | No—does not control indexing |
| Use | Tool | Example |
|---|---|---|
| Path-level (whole directory) | robots.txt | Disallow: /admin/, Disallow: /api/, Disallow: /staging/ |
| Page-level (specific pages) | noindex meta / X-Robots-Tag | Login, signup, thank-you, 404, legal. See indexing for full list |
| Critical | Do NOT block in robots.txt | Pages that use noindex—crawlers must access the page to read the directive |
Paths to block in robots.txt: /admin/, /api/, /staging/, temp files. Paths to use noindex (allow crawl): /login/, /signup/, /thank-you/, etc.—see indexing.
| Item | Requirement |
|---|---|
| Path | Site root: https://example.com/robots.txt |
| Encoding | UTF-8 plain text |
| Standard | RFC 9309 (Robots Exclusion Protocol) |
| Directive | Purpose | Example |
|---|---|---|
User-agent: | Target crawler | User-agent: Googlebot, User-agent: * |
Disallow: | Block path prefix | Disallow: /admin/ |
Allow: | Allow path (can override Disallow) | Allow: /public/ |
Sitemap: | Declare sitemap absolute URL | Sitemap: https://example.com/sitemap.xml |
Clean-param: | Strip query params (Yandex) | See below |
Different directives use different path formats — a common source of errors:
| Directive | Format | Correct | Wrong |
|---|---|---|---|
| Disallow / Allow | Root-relative path only (starts with /) | Disallow: /admin/ | Disallow: https://example.com/admin/ |
| Sitemap | Absolute URL only | Sitemap: https://example.com/sitemap.xml | Sitemap: /sitemap.xml |
Wildcards: * matches any character sequence (Disallow: /tmp/*). $ marks exact URL ending (Allow: /news/.html$).
Priority: More specific paths take precedence. Allow: /shop/shoes/ overrides Disallow: /shop/. Path matching is case-sensitive: Disallow: /PDF/ does not match /pdf/.
| Do not block | Reason |
|---|---|
| CSS, JS, images | Google needs them to render pages; blocking breaks indexing |
/_next/ (Next.js) | Breaks CSS/JS loading; static assets in GSC "Crawled - not indexed" is expected. See indexing |
| Pages that use noindex | Crawlers must access the page to read the noindex directive; blocking in robots.txt prevents that |
Only block: paths that don't need crawling: /admin/, /api/, /staging/, temp files.
robots.txt is effective for all measured AI crawlers. Set rules per user-agent; check each vendor's docs for current tokens.
| User-agent | Purpose | Typical | Notes |
|---|---|---|---|
| OAI-SearchBot | ChatGPT search | Allow | Respects robots.txt |
| GPTBot | OpenAI training | Disallow | Respects robots.txt; shares crawl data with OAI-SearchBot if both allowed |
| ChatGPT-User | User-initiated browsing | N/A | No longer respects robots.txt (Dec 2025); use server-side controls instead |
| Claude-SearchBot | Claude search | Allow | Respects robots.txt |
| Claude-User | Anthropic user-initiated browsing | Allow | Respects robots.txt (unlike ChatGPT-User) |
| ClaudeBot | Anthropic training | Disallow | Respects robots.txt |
Deprecated:
anthropic-ai— retired by Anthropic, replaced by ClaudeBot / Claude-User / Claude-SearchBot. References toanthropic-aiin robots.txt have no effect. | PerplexityBot | Perplexity search | Allow | Respects robots.txt | | Google-Extended | Gemini training | Disallow | Respects robots.txt | | CCBot | Common Crawl (LLM training) | Disallow | Respects robots.txt | | Bytespider | ByteDance | Disallow | Respects robots.txt | | Meta-ExternalAgent | Meta | Disallow | Respects robots.txt | | AppleBot | Apple (Siri, Spotlight); renders JS | Allow for indexing | Respects robots.txt |
Allow vs Disallow: Allow search/indexing bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot); Disallow training-only bots (GPTBot, ClaudeBot, CCBot) if you don't want content used for model training.
Important — ChatGPT-User exemption: As of December 2025, ChatGPT-User no longer respects robots.txt directives. OpenAI considers it a proxy for human-initiated browsing. If you need to block it, use server-side controls (WAF rules, IP rate-limiting), not robots.txt. See site-crawlability for AI crawler optimization (SSR, URL management).
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&ref&fbclid&gclid
まだレビューはありません。使ってみた感想をお寄せください。
ログインなしで書き込めます。使ってみた感想、質問、動かなかった環境などを気軽にどうぞ。
まだコメントはありません。最初のコメントを書いてみませんか。
When the user wants to create, optimize, or audit 404 error pages. Also use when the user mentions "404 page," "404 error," "error page," "page not found," "broken link page," "404 design," "custom 404," "404 redirect," "404 page UX," or "404 recovery." For sitewide page planning, use website-structure.
日本語の概要は準備中です。原文の説明を表示しています。
When the user wants to create, optimize, or audit About page content. Also use when the user mentions "about page," "about us," "company story," "our team," "about section," "company overview," "brand story," "team page," or "who we are." For sitewide page planning, use website-structure.
日本語の概要は準備中です。原文の説明を表示しています。
When the user wants to plan, implement, or optimize affiliate marketing strategy. Also use when the user mentions "affiliate marketing," "affiliate program strategy," "CPS model," "affiliate recruitment," "commission structure," "affiliate partners," "affiliate network," "affiliate tracking," "affiliate commission," or "partner marketing." For affiliate page, use affiliate-page-generator.
日本語の概要は準備中です。原文の説明を表示しています。
When the user wants to create, optimize, or audit affiliate program page content. Also use when the user mentions "affiliate program," "affiliate page," "partner program," "referral program page," "affiliate landing," "partner landing," "commission page," or "affiliate signup." For affiliate program strategy, use affiliate-marketing.
日本語の概要は準備中です。原文の説明を表示しています。
When the user wants to track AI search traffic in GA4 or GSC. Also use when the user mentions "AI traffic," "ChatGPT referral," "Perplexity traffic," "AI Overviews," "GA4 AI sources," "AI search analytics," "track AI referrals," "AI search traffic," "Claude traffic," or "how to track AI traffic." For AI SEO strategy, use generative-engine-optimization.
日本語の概要は準備中です。原文の説明を表示しています。
When the user wants to create, optimize, or audit alternatives or comparison content (page or blog article). Also use when the user mentions "alternatives page," "alternatives listicle," "X alternatives," "competitor comparison," "vs page," "compare page," "best alternatives to X," "switch from X," "competitor brand traffic," "brand keyword ads," or "intercept competitor search." For competitor research, use competitor-research.
日本語の概要は準備中です。原文の説明を表示しています。