Results for “crawl”

Any live-web task via the Firecrawl CLI — including ordinary web research: searching the web, reading or extracting pages, gathering sources, discovering site URLs, bulk extraction, downloading a site, change alerts, or pages needing clicks/login — web only; local files route to firecrawl-parse. For US legal or regulatory questions use gov; for papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.

The English summary is being prepared. Showing the original description.

Browsertrix Crawler is a browser-based crawling system from Webrecorder for high-fidelity site capture, QA replay analysis, and configurable crawl behavior. It runs complex crawls in a single Docker container and uses Puppeteer plus Chrome DevTools Protocol under the hood.

The English summary is being prepared. Showing the original description.

Search, scrape, and interact with the web via the Firecrawl CLI. Use this skill whenever the user wants to search the web, find articles, research a topic, look something up online, scrape a webpage, grab content from a URL, get data from a website, crawl documentation, or interact with pages that need clicks or logins. Also use when they say "fetch this page", "pull the content from", "get the page at https://", or reference external websites. This provides real-time web search with full page content and interact capabilities — beyond typical built-in browsing without the Firecrawl CLI. Do NOT trigger for local file operations, git commands, deployments, or code editing tasks.

The English summary is being prepared. Showing the original description.

Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.

The English summary is being prepared. Showing the original description.

Analyze server log files for crawl budget insights. Reads Apache Combined/Common or Nginx access logs locally (no external calls). Classifies bot vs user traffic, identifies top crawled URLs, crawl frequency by path, and crawl budget concerns. Use when user says "log analysis", "crawl budget", "server logs", "bot traffic", "crawl frequency", "access log", "analyze logs".

The English summary is being prepared. Showing the original description.

Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.

The English summary is being prepared. Showing the original description.

Convert existing scripts, CLI tools, or Crawlee projects into reusable Apify Actors, programs packaged as Docker images. Define JSON inputs and outputs, integrate the SDK, test locally, and deploy to Apify.

• Turning an existing script or CLI tool into an Actor
• Migrating a Crawlee project to Apify
• Defining JSON inputs and structured outputs

Equip NanoClaw agents to convert attached office documents into Markdown locally using Firecrawl AnyDoc. Agents can read Word, PowerPoint, Excel, and text-based PDF files without uploading them to an external parsing service.

• Reading and summarizing attached office documents
• Converting documents without uploading them to an external service
• Extracting text from text-based PDFs

Audit a website’s technical foundations for search engines and AI services. Check crawler access, indexing, security, mobile usability, speed, and server-rendered content, then produce a scored Markdown report with prioritized findings.

• Check whether AI crawlers can access key pages
• Find content missing from the initial HTML
• Review page speed and mobile usability

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

The English summary is being prepared. Showing the original description.

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.

The English summary is being prepared. Showing the original description.

Triage technical SEO findings from Ahrefs Site Audit (and similar crawlers) by SEO impact, not just severity. Use this skill when reviewing crawl results, prioritizing technical fixes, scoping a technical SEO sprint, or after running any site-wide crawl. Triggers on site audit results, technical fix list, crawl errors, technical SEO triage, prioritize technical issues, what should we fix first, broken links, redirect chains. Also triggers when a long list of crawler issues is creating decision paralysis.

The English summary is being prepared. Showing the original description.

Find and synthesize research papers, whitepapers, PDFs, technical reports, and academic sources with Firecrawl Research, using semantic paper search, related-paper expansion, and in-body verification over Firecrawl's paper index — largely biomedical and life-science literature from PubMed, bioRxiv, and medRxiv, plus arXiv preprints in CS, physics, and math. Use when the user wants a literature review, systematic review, survey of studies, paper summary, research landscape, or sourced synthesis from scholarly and industry publications, including clinical, drug, gene, disease, epidemiology, and public-health topics. Prefer this over a general web-research workflow whenever the evidence base is published papers rather than web pages.

The English summary is being prepared. Showing the original description.

Produce an intensive, cited analytical report: executive summary, multi-angle findings, contrarian views, open questions, and full sources. Use only when the user needs rigorous synthesis of a complex topic (scientific, technical, policy, or market-analytical) that cannot be answered with a short search, and wants a formal written report, not a recommendation list. Do not use for product picks, top-N lists, quick lookups, or routine "find out about X" tasks. If the request does not clearly need this kind of report, do not use this skill. Do not use for a literature review over published papers. This skill collects evidence from the open web. A request for the literature on a biomedical, clinical, life-science, or other scientific topic — papers, studies, trials, preprints — belongs to firecrawl-research-papers, which queries Firecrawl's paper index (PubMed, bioRxiv, medRxiv, arXiv) instead of searching websites.

The English summary is being prepared. Showing the original description.

Find and qualify new prospects with Firecrawl Alexandria people and company data: people by role, seniority, company, industry, size, location or technology, target accounts that match criteria, or the right contacts at companies, returned as a ranked, sourced table or CSV. Also extracts lead lists from a database or directory the user names, and refreshes a prospect list with only new matches. Use when the user wants a lead list, prospects, target accounts or the people to reach out to at companies that fit criteria. Not for enriching people or companies the user already supplied (use firecrawl-lead-research).

The English summary is being prepared. Showing the original description.

You are an expert in Crawl4AI, the open-source web crawler built for AI applications. You help developers extract clean, structured data from websites for LLM training, RAG pipelines, and content analysis — with automatic markdown conversion, JavaScript rendering, CSS-based extraction, LLM-powered structured extraction, and session management for multi-page crawling.

The English summary is being prepared. Showing the original description.

Full-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site structure", "discover pages", "JS rendering", or needs site-wide analysis.

The English summary is being prepared. Showing the original description.

Find the published business contacts of companies with Firecrawl, whether the user names them or describes them by type and place: press and media, sales, partnerships and support inboxes, shared business phone numbers and official contact forms, taken from the companies' own websites. Use when the user asks how to contact a company or wants published business emails, phone numbers or press, sales, support or partnership contacts, since contact pages change and an address must come from the page, not memory. Not for named people's details (use firecrawl-lead-research) or finding people to reach out to (use firecrawl-lead-gen).

The English summary is being prepared. Showing the original description.

Enrich and research the people and companies the user names or supplies with Firecrawl Alexandria data: background on a person, a professional profile URL to look up, whether someone still holds a role, missing columns in a table or CSV of people or companies, firmographics for company domains (industry, headcount, funding, location), and pre-call or meeting briefs. Use whenever the request is about specific named people or companies, even without the word enrich, since roles, headcount and funding change faster than training data. Not for finding new prospects (use firecrawl-lead-gen) or a company's published press, sales or support contacts (use firecrawl-business-contacts).

The English summary is being prepared. Showing the original description.

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.

The English summary is being prepared. Showing the original description.

Scrape web pages to clean markdown using Firecrawl v2 — handles JS-heavy pages, site crawls, URL mapping, document parsing (PDF/DOCX/XLSX), LLM-powered extraction, autonomous agent scraping, and post-scrape browser interaction (Interact API). Free keyless mode available for search/scrape/parse/interact (no API key needed). Prefer over WebFetch for quality and completeness. Triggers on scrape URL, fetch page, crawl site, extract content, parse document, web to markdown, DeepWiki, Firecrawl.

The English summary is being prepared. Showing the original description.

End-to-end SEO operations for B2B SaaS organic visibility. Use this skill when you need keyword research, technical SEO audits, content optimization, link building strategy, international expansion, AI/AEO optimization, schema markup, Core Web Vitals improvements, and organic traffic growth planning. Also triggers on: SEO, keyword research, can we rank for this keyword, which keywords are actually worth it, qualify these keywords, check the SERPs, keyword shortlist, too many keywords, technical SEO, link building, content optimization, international SEO, hreflang, x-default, multilingual site, translated pages not ranking, our German site gets no traffic, language switcher, auto-redirect by country, machine translation SEO, content parity, ccTLD vs subfolder, AI search, AEO, GEO, Core Web Vitals, schema markup, organic traffic, G2 listing, Capterra, TrustRadius, review-platform category page, best software roundup, listicle placement, directory submissions, do we need a Google Business Profile in [country], local SEO with no office, Google Business Profile eligibility, virtual office address SEO, coworking address business profile, business profile suspended, NAP consistency, local citations for SaaS, LocalBusiness schema, server log analysis, log file analysis, what is Googlebot crawling, is Googlebot hitting errors, verify Googlebot, fake Googlebot, crawl budget, do we have a crawl budget problem, Crawl Stats report, new pages not getting crawled.

The English summary is being prepared. Showing the original description.

Get structured data from catalogued providers instead of scraping pages. Firecrawl Alexandria is a library of official APIs, licensed publishers, and Firecrawl indexes (financial series, company data, government contracts and law, podcasts, listings, research). Use this skill when the user needs records, listings, prices, filings, transcripts, or a dataset, says "is there an API for", "find a data source for", "pull the records", "get the dataset", or when assembling the answer by scraping would take many pages that one provider call returns directly. Discover with search, inspect with find_tools, execute with scrape.

The English summary is being prepared. Showing the original description.

Search, scrape, and interact with the web via the Firecrawl CLI. Use this skill whenever the user wants to search the web, find articles, research a topic, look something up online, scrape a webpage, grab content from a URL, get data from a website, crawl documentation, download a site, or interact with pages that need clicks or logins. Also use when they say "fetch this page", "pull the content from", "get the page at https://", or reference external websites. This provides real-time web search with full page content and interact capabilities — beyond what Grok can do natively with built-in tools. Do NOT trigger for local file operations, git commands, deployments, or code editing tasks.

The English summary is being prepared. Showing the original description.