「crawl」の検索結果

Analyze server log files for crawl budget insights. Reads Apache Combined/Common or Nginx access logs locally (no external calls). Classifies bot vs user traffic, identifies top crawled URLs, crawl frequency by path, and crawl budget concerns. Use when user says "log analysis", "crawl budget", "server logs", "bot traffic", "crawl frequency", "access log", "analyze logs".

日本語の概要は準備中です。原文の説明を表示しています。

Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.

日本語の概要は準備中です。原文の説明を表示しています。

Integrate Firecrawl `/scrape` into product code for single-page extraction. Use when an app already has a URL and needs markdown, HTML, links, screenshots, metadata, or structured page output. Prefer this skill over broader crawl patterns when the feature is page-level.

日本語の概要は準備中です。原文の説明を表示しています。

Full-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site structure", "discover pages", "JS rendering", or needs site-wide analysis.

日本語の概要は準備中です。原文の説明を表示しています。

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

日本語の概要は準備中です。原文の説明を表示しています。

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.

日本語の概要は準備中です。原文の説明を表示しています。

Triage technical SEO findings from Ahrefs Site Audit (and similar crawlers) by SEO impact, not just severity. Use this skill when reviewing crawl results, prioritizing technical fixes, scoping a technical SEO sprint, or after running any site-wide crawl. Triggers on site audit results, technical fix list, crawl errors, technical SEO triage, prioritize technical issues, what should we fix first, broken links, redirect chains. Also triggers when a long list of crawler issues is creating decision paralysis.

日本語の概要は準備中です。原文の説明を表示しています。

Full-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site structure", "discover pages", "JS rendering", or needs site-wide analysis.

日本語の概要は準備中です。原文の説明を表示しています。

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.

日本語の概要は準備中です。原文の説明を表示しています。

Find and synthesize research papers, whitepapers, PDFs, technical reports, and academic sources with Firecrawl Research, using semantic paper search, related-paper expansion, and in-body verification over Firecrawl's paper index — largely biomedical and life-science literature from PubMed, bioRxiv, and medRxiv, plus arXiv preprints in CS, physics, and math. Use when the user wants a literature review, systematic review, survey of studies, paper summary, research landscape, or sourced synthesis from scholarly and industry publications, including clinical, drug, gene, disease, epidemiology, and public-health topics. Prefer this over a general web-research workflow whenever the evidence base is published papers rather than web pages.

日本語の概要は準備中です。原文の説明を表示しています。

Run outcome-focused Firecrawl workflows that produce deliverables such as research reports, literature reviews over published papers, SEO audits, QA reports, lead lists, knowledge bases, website design systems, and other structured web-data artifacts. Use when the user wants Firecrawl to complete a business, marketing, product, or creative workflow rather than merely scrape a page or integrate API calls into code.

日本語の概要は準備中です。原文の説明を表示しています。

Produce an intensive, cited analytical report: executive summary, multi-angle findings, contrarian views, open questions, and full sources. Use only when the user needs rigorous synthesis of a complex topic (scientific, technical, policy, or market-analytical) that cannot be answered with a short search, and wants a formal written report, not a recommendation list. Do not use for product picks, top-N lists, quick lookups, or routine "find out about X" tasks. If the request does not clearly need this kind of report, do not use this skill. Do not use for a literature review over published papers. This skill collects evidence from the open web. A request for the literature on a biomedical, clinical, life-science, or other scientific topic — papers, studies, trials, preprints — belongs to firecrawl-research-papers, which queries Firecrawl's paper index (PubMed, bioRxiv, medRxiv, arXiv) instead of searching websites.

日本語の概要は準備中です。原文の説明を表示しています。

You are an expert in Crawl4AI, the open-source web crawler built for AI applications. You help developers extract clean, structured data from websites for LLM training, RAG pipelines, and content analysis — with automatic markdown conversion, JavaScript rendering, CSS-based extraction, LLM-powered structured extraction, and session management for multi-page crawling.

日本語の概要は準備中です。原文の説明を表示しています。

Full-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site structure", "discover pages", "JS rendering", or needs site-wide analysis.

日本語の概要は準備中です。原文の説明を表示しています。

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.

日本語の概要は準備中です。原文の説明を表示しています。

Scrape web pages to clean markdown using Firecrawl v2 — handles JS-heavy pages, site crawls, URL mapping, document parsing (PDF/DOCX/XLSX), LLM-powered extraction, autonomous agent scraping, and post-scrape browser interaction (Interact API). Free keyless mode available for search/scrape/parse/interact (no API key needed). Prefer over WebFetch for quality and completeness. Triggers on scrape URL, fetch page, crawl site, extract content, parse document, web to markdown, DeepWiki, Firecrawl.

日本語の概要は準備中です。原文の説明を表示しています。

End-to-end SEO operations for B2B SaaS organic visibility. Use this skill when you need keyword research, technical SEO audits, content optimization, link building strategy, international expansion, AI/AEO optimization, schema markup, Core Web Vitals improvements, and organic traffic growth planning. Also triggers on: SEO, keyword research, can we rank for this keyword, which keywords are actually worth it, qualify these keywords, check the SERPs, keyword shortlist, too many keywords, technical SEO, link building, content optimization, international SEO, hreflang, x-default, multilingual site, translated pages not ranking, our German site gets no traffic, language switcher, auto-redirect by country, machine translation SEO, content parity, ccTLD vs subfolder, AI search, AEO, GEO, Core Web Vitals, schema markup, organic traffic, G2 listing, Capterra, TrustRadius, review-platform category page, best software roundup, listicle placement, directory submissions, do we need a Google Business Profile in [country], local SEO with no office, Google Business Profile eligibility, virtual office address SEO, coworking address business profile, business profile suspended, NAP consistency, local citations for SaaS, LocalBusiness schema, server log analysis, log file analysis, what is Googlebot crawling, is Googlebot hitting errors, verify Googlebot, fake Googlebot, crawl budget, do we have a crawl budget problem, Crawl Stats report, new pages not getting crawled.

日本語の概要は準備中です。原文の説明を表示しています。

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.

日本語の概要は準備中です。原文の説明を表示しています。

Get structured data from catalogued providers instead of scraping pages. Firecrawl Alexandria is a library of official APIs, licensed publishers, and Firecrawl indexes (financial series, company data, government contracts and law, podcasts, listings, research). Use this skill when the user needs records, listings, prices, filings, transcripts, or a dataset, says "is there an API for", "find a data source for", "pull the records", "get the dataset", or when assembling the answer by scraping would take many pages that one provider call returns directly. Discover with search, inspect with find_tools, execute with scrape.

日本語の概要は準備中です。原文の説明を表示しています。

Search, scrape, and interact with the web via the Firecrawl CLI. Use this skill whenever the user wants to search the web, find articles, research a topic, look something up online, scrape a webpage, grab content from a URL, get data from a website, crawl documentation, download a site, or interact with pages that need clicks or logins. Also use when they say "fetch this page", "pull the content from", "get the page at https://", or reference external websites. This provides real-time web search with full page content and interact capabilities — beyond what Grok can do natively with built-in tools. Do NOT trigger for local file operations, git commands, deployments, or code editing tasks.

日本語の概要は準備中です。原文の説明を表示しています。

Full-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site structure", "discover pages", "JS rendering", or needs site-wide analysis.

日本語の概要は準備中です。原文の説明を表示しています。

Integrate Firecrawl `/interact` into product code for dynamic pages and browser actions after scraping. Use when a feature needs clicks, form fills, pagination, authentication-aware flows, or other multi-step interactions that plain `/scrape` cannot complete.

日本語の概要は準備中です。原文の説明を表示しています。

Integrate Firecrawl `/search` into product code and agent workflows. Use when an app needs discovery before extraction, when the feature starts with a query instead of a URL, or when the system should search the web and optionally hydrate result content.

日本語の概要は準備中です。原文の説明を表示しています。

XML sitemap audit — find and fix the sitemap problems that quietly waste crawl budget and slow indexing. Discovers the sitemap (robots.txt, /sitemap.xml, sitemap index), validates structure and size limits, and cross-checks the URLs it lists against reality: non-200 / redirected / noindex / canonicalized-away URLs that shouldn't be in a sitemap, plus indexable pages that are missing from it. Reviews lastmod accuracy, sitemap-index organization, and robots.txt reference. Use this skill whenever the user asks about sitemaps, sitemap errors in Search Console, "sitemap couldn't fetch / has errors", crawl budget, pages not getting indexed, or whether their sitemap is clean. Trigger on: "sitemap", "sitemap.xml", "XML sitemap", "sitemap errors", "sitemap audit", "couldn't fetch sitemap", "crawl budget", "pages not indexed sitemap", "sitemap index", "lastmod", "robots.txt sitemap", or any sitemap/crawl-coverage question. For a full-site SEO audit use /seo-analysis; for broken links use /broken-link-checker.

日本語の概要は準備中です。原文の説明を表示しています。