Results for “web crawler”

Complete knowledge domain for Firecrawl v2 API - web scraping and crawling that converts websites into LLM-ready markdown or structured data. Use when: scraping websites, crawling entire sites, extracting web content, converting HTML to markdown, building web scrapers, handling dynamic JavaScript content, bypassing anti-bot protection, extracting structured data from web pages, or when encountering "content not loading", "JavaScript rendering issues", or "blocked by bot detection". Keywords: firecrawl, firecrawl api, web scraping, web crawler, scrape website, crawl website, extract content, html to markdown, site crawler, content extraction, web automation, firecrawl-py, firecrawl-js, llm ready data, structured data extraction, bot bypass, javascript rendering, scraping api, crawling api, map urls, batch scraping

The English summary is being prepared. Showing the original description.

Crawl linked pages within a chosen website scope and collect clean text, Markdown, or HTML with page metadata. Use the resulting records as source material for AI search systems or chatbot knowledge bases.

• Collect documentation as Markdown
• Prepare source material for a chatbot knowledge base
• Gather blog content within a chosen scope

Check whether AI crawlers, the bots that collect web content for AI services, can access a website. Get a crawler access table, an assessment score, and recommendations based on robots.txt, page tags, HTTP headers, and rendering requirements.

• Checking which AI crawlers can access a website
• Reviewing unintended blocks in robots.txt
• Investigating content that requires JavaScript to display

Crawl websites (news, blogs, papers, GitHub, product docs, RSS feeds) into a fixed-schema JSONL file, then create a dataset and a searchable application in Viking AI Search. Supports one-time crawl and scheduled recurring crawl with automatic incremental sync.

The English summary is being prepared. Showing the original description.

Extract website content into Markdown, HTML, or plain text with Scrapling. Handle JavaScript-rendered pages and anti-bot challenges, or write Python crawlers that follow links, collect structured data, and resume interrupted runs.

• Save article text as Markdown
• Extract content loaded by JavaScript
• Retrieve pages with anti-bot protections

Audit a website’s technical foundations for search engines and AI services. Check crawler access, indexing, security, mobile usability, speed, and server-rendered content, then produce a scored Markdown report with prioritized findings.

• Check whether AI crawlers can access key pages
• Find content missing from the initial HTML
• Review page speed and mobile usability

Search the public web and crawl URLs via the Unique AI Platform's `unique-cli web-search` command, with automatic per-turn citation tracking so cited web facts render as clickable external reference chips and `<sup>N</sup>` footnotes on the Unique platform. ALWAYS use this skill when the user asks to search the public web, research a topic online, look up news/articles/regulations, fetch the contents of one or more URLs, or perform any "two-phase" web research (search → review snippets → crawl selected URLs for full content). Routes through the Unique AI Platform's `/web-search-api` endpoints, so the engine (Google, Brave, Tavily, Jina, Firecrawl, …) and crawler are resolved server-side from the current environment — you never need to manage engine API keys yourself. Use this instead of fabricating answers from training data whenever fresh / source-cited information is needed. Do NOT use this skill for the internal knowledge base (`unique-cli search`) or for arbitrary tool execution (`unique-cli mcp`).

The English summary is being prepared. Showing the original description.

Audit a website for visibility in AI search, including how easily its content can be quoted, technical access, content quality, and structured data. Receive a weighted score out of 100 and a prioritized improvement plan.

• Assessing a website’s readiness for AI search
• Finding crawler access and structured data issues
• Reviewing content credibility and ease of quotation

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

The English summary is being prepared. Showing the original description.

Audit a public website for search visibility and AI readability, then prepare prioritized fixes, configuration files, and revised copy. For connected stores, review product listings and apply approved batches of updates to titles, descriptions, and SEO fields.

• Audit a website for search and AI readability
• Prepare crawler settings and machine-readable business data
• Rewrite page titles, descriptions, and content

Assess a website’s readiness for AI search alongside traditional SEO. Review content citation readiness, crawler access, brand mentions, technical foundations, and structured data, then generate scores, prioritized recommendations, and client reports.

• Audit a website’s readiness for AI search
• Assess a page’s suitability for AI citation
• Check AI crawler access and generate llms.txt

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

The English summary is being prepared. Showing the original description.

"Implement a web crawler pipeline covering URL discovery, fetching, parsing, and storage. Use this skill when the user needs to build a site crawler, audit website structure, or collect web data systematically — even if they say 'scrape a website', 'crawl all pages', or 'site audit spider'.".

The English summary is being prepared. Showing the original description.

Audit the technical factors that affect how search engines discover and index a website. Check crawl access, URLs, security, mobile pages, loading performance, and JavaScript rendering, then report measured results with prioritized fixes.

• Investigate why pages are missing from search results
• Check technical SEO before a launch or site migration
• Review mobile content and page performance

Browsertrix Crawler is a browser-based crawling system from Webrecorder for high-fidelity site capture, QA replay analysis, and configurable crawl behavior. It runs complex crawls in a single Docker container and uses Puppeteer plus Chrome DevTools Protocol under the hood.

The English summary is being prepared. Showing the original description.

Audit a website for Google AdSense application readiness or ongoing ad-serving issues. Check content, navigation, crawler access, and disclosures against official requirements, then receive evidence-based findings, specific fixes, and a complete checklist with unresolved questions clearly marked.

• Checking a site before applying for AdSense
• Investigating issues after an AdSense rejection
• Reviewing policy risks on a site that already serves ads

Build a complete web scraping Actor with Crawlee and deploy to Apify. Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform. Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".

The English summary is being prepared. Showing the original description.

You are an expert in Crawl4AI, the open-source web crawler built for AI applications. You help developers extract clean, structured data from websites for LLM training, RAG pipelines, and content analysis — with automatic markdown conversion, JavaScript rendering, CSS-based extraction, LLM-powered structured extraction, and session management for multi-page crawling.

The English summary is being prepared. Showing the original description.

Deploys intelligent web scraping actors on the Apify platform using the Apify SDK with RequestQueue and Dataset APIs. Handles dynamic content via Apify CheerioCrawler and PlaywrightCrawler with automatic scaling.

The English summary is being prepared. Showing the original description.

Comprehensive SEO analysis for any website or business type. Performs full site audits, single-page deep analysis, technical SEO checks (crawlability, indexability, Core Web Vitals with INP), schema markup detection/validation/generation, content quality assessment (E-E-A-T framework per Dec 2025 update extending to all competitive queries), image optimization, sitemap analysis, and Generative Engine Optimization (GEO) for AI Overviews, ChatGPT, and Perplexity citations. Analyzes AI crawler accessibility (GPTBot, ClaudeBot, PerplexityBot), llms.txt compliance, brand mention signals, and passage-level citability. Industry detection for SaaS, e-commerce, local business, publishers, agencies. Triggers on: "SEO", "audit", "schema", "Core Web Vitals", "sitemap", "E-E-A-T", "AI Overviews", "GEO", "technical SEO", "content quality", "page speed", "structured data".

The English summary is being prepared. Showing the original description.

Comprehensive SEO analysis for any website or business type. Orchestrates 52 sub-skills (52 active): full site audits, technical SEO (CWV/INP), schema, E-E-A-T content quality, images, sitemaps, GEO for AI Overviews, llms.txt generation, AI crawler audit, plus live GSC and Ahrefs MCP data. GSC: overview, drops, opportunities, brand-vs-nonbrand, cannibalization, compare, content-decay, index-issues, new-keywords. Ahrefs: overview, backlinks, keywords, competitors, content-gap, broken-links, new-links, anchor-analysis, dr-history, top-pages. Cross-MCP: serp, content-brief, brand-radar, site-audit-pro, report. Local: log-analysis, ai-content-check, internal-links, local, migration-check, obsidian-kb. AI Readability: llms-txt, robots-ai. Performance: lighthouse-audit, core-web-vitals, page-experience. Quality & AI Search: rich-results, ai-optimization, eeat-audit, helpful-content. Triggers on: "SEO", "audit", "schema", "Core Web Vitals", "sitemap", "E-E-A-T", "AI Overviews", "GEO", "technical SEO", "content quality", "structured data", "GSC", "Ahrefs", "backlinks", "keywords", "search console", "domain rating", "local SEO", "migration", "lighthouse", "pagespeed", "LCP", "INP", "CLS", "page experience", "interstitials", "mobile friendly", "rich results", "rich snippets", "schema eligibility", "AI optimization", "AI search readiness", "EEAT", "trust signals", "helpful content", "people-first", "content quality gate".

The English summary is being prepared. Showing the original description.

Audit whether a website can be found, crawled, and cited by AI answer engines such as ChatGPT Search, Perplexity, Google AI Overviews, and Microsoft Copilot. Use when someone asks why their brand is missing from AI answers, whether AI crawlers can read their site, how to get cited by ChatGPT or Perplexity, or asks for a GEO or AEO (generative / answer engine optimization) review. Produces a citation baseline across buyer-intent prompts, a crawler-access check, a citability review of named pages, and a ranked fix list. Not for keyword rank tracking, paid search, or pages behind a login.

The English summary is being prepared. Showing the original description.

WeChat Mini Program development skill for building, debugging, previewing, testing, publishing, and optimizing mini program projects (小程序开发、调试、预览、发布). Covers project structure and config (`project.config.json`, `appid`, `miniprogramRoot`, `tabBar`, routing/navigation, icon assets), WeChat Developer Tools Nightly workflows (`wechatide` CLI, WeChat IDE Skills/MCP), `miniprogram-ci` preview/upload, console/network debugging, message push (消息推送) and customer-service auto-reply (客服消息), mini program SEO / search indexing (小程序搜索优化、页面收录、搜索推广、mpcrawler), and CloudBase integration (`wx.cloud`, 腾讯云开发, 云开发) when explicitly used. Use when users create, develop, modify, debug, preview, deploy, publish, or promote WeChat Mini Programs. NOT for Web frontend (use web-development), pure backend services (use cloudrun-development / cloud-functions), or UI-design-only tasks (use ui-design).

The English summary is being prepared. Showing the original description.

Apify SDK is the official JavaScript SDK for building Actors, crawlers, and data extraction workflows on Apify. It gives agents a structured way to run scraping jobs, store outputs, manage inputs, and combine crawler logic with browser automation when needed.

The English summary is being prepared. Showing the original description.