“crawl”的搜索结果

帮助应用完成Firecrawl初始接入，涵盖浏览器认证、接口密钥设置和SDK选择，并在新项目或现有项目中发起首次接口请求、验证集成是否正常运行。

• 在新应用中接入Firecrawl
• 配置接口密钥
• 选择适合开发语言的SDK

指导将Firecrawl的网页搜索、页面内容获取、结构化信息提取及浏览器操作集成到应用或智能体中，涵盖功能选择、配置、实现和实际请求验证。

• 为应用添加网页搜索
• 从已知网址提取结构化数据
• 将点击和表单填写纳入流程

帮助将Firecrawl的`/search`集成到应用或智能体流程中，从搜索词找到候选网页、选择信息来源，并决定是否进一步获取页面正文。

• 为带出处的回答寻找信息来源
• 探索企业、竞争对手或主题
• 筛选值得深入阅读的网页

帮助将Firecrawl的`/scrape`集成到应用中，从已知网址获取单页正文、链接、图片等信息，并设置复用数据的时效限制。

• 导入已知网页以便生成摘要
• 从企业或产品页面补充信息
• 提取价格、更新记录和文档

帮助将Firecrawl的`/interact`集成到应用中，通过点击、填写表单和翻页获取网页信息，并设计读取页面后保留浏览器状态的流程。

• 获取点击后才显示的内容
• 操作搜索表单和筛选条件
• 获取分布在多页的结果

Crawl websites through Firecrawl to collect URLs, page content, metadata, and links. Extract content that appears through JavaScript, map site structure, and gather material for broken link checks and broader SEO analysis.

中文简介正在准备中。当前显示英语简介。

• Mapping site URLs and comparing them with an XML sitemap
• Collecting content from a blog or product section
• Extracting content rendered through JavaScript

Crawl websites (news, blogs, papers, GitHub, product docs, RSS feeds) into a fixed-schema JSONL file, then create a dataset and a searchable application in Viking AI Search. Supports one-time crawl and scheduled recurring crawl with automatic incremental sync.

中文简介正在准备中。当前显示原文说明。

Complete knowledge domain for Firecrawl v2 API - web scraping and crawling that converts websites into LLM-ready markdown or structured data. Use when: scraping websites, crawling entire sites, extracting web content, converting HTML to markdown, building web scrapers, handling dynamic JavaScript content, bypassing anti-bot protection, extracting structured data from web pages, or when encountering "content not loading", "JavaScript rendering issues", or "blocked by bot detection". Keywords: firecrawl, firecrawl api, web scraping, web crawler, scrape website, crawl website, extract content, html to markdown, site crawler, content extraction, web automation, firecrawl-py, firecrawl-js, llm ready data, structured data extraction, bot bypass, javascript rendering, scraping api, crawling api, map urls, batch scraping

中文简介正在准备中。当前显示原文说明。

Crawl linked pages within a chosen website scope and collect clean text, Markdown, or HTML with page metadata. Use the resulting records as source material for AI search systems or chatbot knowledge bases.

中文简介正在准备中。当前显示英语简介。

• Collect documentation as Markdown
• Prepare source material for a chatbot knowledge base
• Gather blog content within a chosen scope

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

中文简介正在准备中。当前显示原文说明。

When the user wants to improve crawlability, fix orphan pages, or optimize site structure for search engines. Also use when the user mentions "crawlability," "crawl budget," "orphan pages," "internal links," "site structure," "site crawlability," "infinite scroll," "pagination," "masonry SEO," "AI crawler optimization," "GPTBot crawlability," "ClaudeBot crawlability," or "content not indexed." For internal links, use internal-links.

中文简介正在准备中。当前显示原文说明。

Check whether AI crawlers, the bots that collect web content for AI services, can access a website. Get a crawler access table, an assessment score, and recommendations based on robots.txt, page tags, HTTP headers, and rendering requirements.

中文简介正在准备中。当前显示英语简介。

• Checking which AI crawlers can access a website
• Reviewing unintended blocks in robots.txt
• Investigating content that requires JavaScript to display

Any live-web task via the Firecrawl CLI — including ordinary web research: searching the web, reading or extracting pages, gathering sources, discovering site URLs, bulk extraction, downloading a site, change alerts, or pages needing clicks/login — web only; local files route to firecrawl-parse. For papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.

中文简介正在准备中。当前显示原文说明。

Find the papers that answer a research query in Firecrawl's research paper index — a corpus of paper abstracts whose largest share is biomedical and life-science literature (PubMed, bioRxiv, medRxiv), alongside arXiv preprints in CS, physics, and math — using semantic search, semantic and structural expansion, and in-body verification. Use this skill for literature-finding and paper-retrieval tasks of any kind, including clinical, biomedical, drug, gene, disease, and other life-science questions, whether the answer is a single paper or a full multi-paper set. The index is reached only through the `firecrawl_research_*` MCP tools or the `firecrawl research` CLI subcommands. Calling `firecrawl_search` with its `categories` option set to `["research"]` is a different feature — it filters ordinary web search to research-affiliated websites (the list includes PubMed, bioRxiv, medRxiv, arXiv, and publisher sites) and returns page results from them, without querying the paper records in this index.

中文简介正在准备中。当前显示原文说明。

Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.

中文简介正在准备中。当前显示原文说明。

Curate Gaia bot crawler branches into real-skill catalog entries and clean up processed bot/* branches. Use when the user says: "process bot branches", "curate the crawl output", "clean up bot proposals", "run bot curation", "merge the crawler results", "review bot/* branches", "ingest crawler data", or explicitly types /gaia-bot-curate. Also triggers when the user asks to review pending `bot/*` branches, turn crawler JSON proposals into registry nodes, decide which crawled skills to accept vs reject, or batch-add skills from an automated crawl run. This skill handles the full pipeline: fetch remote bot branches, triage proposals, run `gaia dev` mutations, validate, open a PR, and delete consumed bot branches.

中文简介正在准备中。当前显示原文说明。

Use when a request needs external sources, current facts, listings, a supplied URL, papers, or library and API documentation. When using Firecrawl for the request, use firecrawl_search to find sources and firecrawl_scrape to read a supplied or discovered URL. Firecrawl also retrieves documents, collects structured data, and tracks website changes. Respect explicit source and tool preferences.

中文简介正在准备中。当前显示原文说明。

Scrape and crawl websites with Firecrawl: scrape a page, start a crawl, check crawl status, map a site, search the web. Trigger phrases: firecrawl, scrape website.

中文简介正在准备中。当前显示原文说明。

Find the papers that answer a research query in Firecrawl's research paper index — a corpus of paper abstracts whose largest share is biomedical and life-science literature (PubMed, bioRxiv, medRxiv), alongside arXiv preprints in CS, physics, and math — using semantic search, semantic and structural expansion, and in-body verification. Use this skill for literature-finding and paper-retrieval tasks of any kind, including clinical, biomedical, drug, gene, disease, and other life-science questions, whether the answer is a single paper or a full multi-paper set. The index is reached only through the `firecrawl_research_*` MCP tools or the `firecrawl research` CLI subcommands. Calling `firecrawl_search` with its `categories` option set to `["research"]` is a different feature — it filters ordinary web search to research-affiliated websites (the list includes PubMed, bioRxiv, medRxiv, arXiv, and publisher sites) and returns page results from them, without querying the paper records in this index.

中文简介正在准备中。当前显示原文说明。

Any live-web task via the Firecrawl CLI — including ordinary web research: searching the web, reading or extracting pages, gathering sources, discovering site URLs, bulk extraction, downloading a site, change alerts, or pages needing clicks/login — web only; local files route to firecrawl-parse. For US legal or regulatory questions use gov; for papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.

中文简介正在准备中。当前显示原文说明。

Fix rsky-pds createRecord/putRecord hanging indefinitely with no error logs. Use when: (1) createSession succeeds but createRecord never returns, (2) No error appears in PDS logs despite request hanging for 60+ seconds, (3) PDS health endpoint works but write operations hang, (4) readiness probes intermittently fail on rsky-pds pod. Root cause: PDS_CRAWLERS env var configures crawler notification URLs that are called via reqwest HTTP client WITHOUT any timeout in crawlers.rs. If the crawler URL is unreachable from the cluster, every write hangs forever. Also: setting PDS_CRAWLERS="" produces [""] (one empty string element) which causes a "builder error" instead — must fully unset the env var.

中文简介正在准备中。当前显示原文说明。

Find the published business contacts of companies with Firecrawl, whether the user names them or describes them by type and place: press and media, sales, partnerships and support inboxes, shared business phone numbers and official contact forms, taken from the companies' own websites. Use when the user asks how to contact a company or wants published business emails, phone numbers or press, sales, support or partnership contacts, since contact pages change and an address must come from the page, not memory. Not for named people's details (use firecrawl-lead-research) or finding people to reach out to (use firecrawl-lead-gen).

中文简介正在准备中。当前显示原文说明。

Enrich and research the people and companies the user names or supplies with Firecrawl Alexandria data: background on a person, a professional profile URL to look up, whether someone still holds a role, missing columns in a table or CSV of people or companies, firmographics for company domains (industry, headcount, funding, location), and pre-call or meeting briefs. Use whenever the request is about specific named people or companies, even without the word enrich, since roles, headcount and funding change faster than training data. Not for finding new prospects (use firecrawl-lead-gen) or a company's published press, sales or support contacts (use firecrawl-business-contacts).

中文简介正在准备中。当前显示原文说明。

Any live-web task via the Firecrawl CLI — including ordinary web research: searching the web, reading or extracting pages, gathering sources, discovering site URLs, bulk extraction, downloading a site, change alerts, or pages needing clicks/login — web only; local files route to firecrawl-parse. For US legal or regulatory questions use gov; for papers use firecrawl-research-index; for library, API, error, or bug questions use firecrawl-developer-index.

中文简介正在准备中。当前显示原文说明。