Results for “crawling”

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

The English summary is being prepared. Showing the original description.

Complete knowledge domain for Firecrawl v2 API - web scraping and crawling that converts websites into LLM-ready markdown or structured data. Use when: scraping websites, crawling entire sites, extracting web content, converting HTML to markdown, building web scrapers, handling dynamic JavaScript content, bypassing anti-bot protection, extracting structured data from web pages, or when encountering "content not loading", "JavaScript rendering issues", or "blocked by bot detection". Keywords: firecrawl, firecrawl api, web scraping, web crawler, scrape website, crawl website, extract content, html to markdown, site crawler, content extraction, web automation, firecrawl-py, firecrawl-js, llm ready data, structured data extraction, bot bypass, javascript rendering, scraping api, crawling api, map urls, batch scraping

The English summary is being prepared. Showing the original description.

Find and fix internal links that lead to missing pages or server errors. Check rendered pages and HTTP responses, then identify URLs to update or remove and explain how broken links affect search engine crawling.

• Checking internal links after moving or deleting pages
• Finding broken navigation and in-text links
• Reviewing how broken links affect search engine crawling

You are an expert in Crawl4AI, the open-source web crawler built for AI applications. You help developers extract clean, structured data from websites for LLM training, RAG pipelines, and content analysis — with automatic markdown conversion, JavaScript rendering, CSS-based extraction, LLM-powered structured extraction, and session management for multi-page crawling.

The English summary is being prepared. Showing the original description.

Web search, content extraction, crawling, and deep research via the Tavily CLI. Use this skill whenever the user wants to search the web, find articles, research a topic, look something up online, extract content from a URL, grab text from a webpage, crawl documentation, download a site's pages, discover URLs on a domain, or conduct in-depth research with citations. Also use when they say "fetch this page", "pull the content from", "get the page at https://", "find me articles about", or reference extracting data from external websites. This provides LLM-optimized web search, content extraction, site crawling, URL discovery, and AI-powered deep research — capabilities beyond what agents can do natively. Do NOT trigger for local file operations, git commands, deployments, or code editing tasks.

The English summary is being prepared. Showing the original description.

Route Tavily web-search, extraction, mapping, crawling, and cited-research requests to the narrowest `tvly` command. Use for command-line Tavily work, installation checks, authentication setup, or choosing among the specialized Tavily skills.

The English summary is being prepared. Showing the original description.

Build or review production-ready Tavily SDK and API integrations for web search, extraction, crawling, mapping, and research. Use when implementing Tavily in an agent, RAG pipeline, or application rather than only running one CLI command.

The English summary is being prepared. Showing the original description.

CLI tool for AI-powered web scraping, data extraction, search, and crawling via ScrapeGraph AI. Use when the user needs to scrape websites, extract structured data from URLs, convert pages to markdown, crawl multi-page sites, search the web for information, automate browser interactions (login, click, fill forms), get raw HTML, discover sitemaps, or generate JSON schemas. Triggers on tasks involving: (1) extracting data from websites, (2) web scraping or crawling, (3) converting webpages to markdown, (4) AI-powered web search with extraction, (5) browser automation, (6) generating output schemas for scraping. The CLI is just-scrape (npm package just-scrape).

The English summary is being prepared. Showing the original description.

Web scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python.

The English summary is being prepared. Showing the original description.

Web search, content extraction, crawling, and deep research via the Tavily CLI. Use this skill whenever the user wants to search the web, find articles, research a topic, look something up online, extract content from a URL, grab text from a webpage, crawl documentation, download a site's pages, discover URLs on a domain, or conduct in-depth research with citations. Also use when they say "fetch this page", "pull the content from", "get the page at https://", "find me articles about", or reference extracting data from external websites. This provides LLM-optimized web search, content extraction, site crawling, URL discovery, and AI-powered deep research — capabilities beyond what agents can do natively. Do NOT trigger for local file operations, git commands, deployments, or code editing tasks.

The English summary is being prepared. Showing the original description.

Review URL parameters used for filters, sorting, pagination, and tracking. Check canonical tags, which identify a preferred page URL, and recommend consistent handling of URL variants to address duplicate content and unnecessary search engine crawling.

• Reviewing filtered and sorted product URLs
• Removing tracking parameters from canonical URLs
• Investigating crawlable URL combinations

Scan links within the same domain for the nofollow attribute, a setting used to qualify links for search engines. List affected pages, link text, and destinations, then guide removal or changes based on the intended purpose.

• Auditing nofollow attributes on internal links
• Reviewing links in navigation, headers, and footers
• Finding nofollow applied broadly by templates

Test REST, GraphQL, and gRPC APIs with Strix using schemas, credentials, and example data. Investigate authorization flaws, excessive data exposure, and other API risks, then verify findings with requests that reproduce the issue.

• Testing an API from its specification
• Checking data access across tenants
• Verifying restrictions on administrator routes

Operate Chrome or Brave using signed-in browser sessions to verify deployed pages, reproduce bugs, inspect page structure, and examine errors or network traffic. An optional macOS bridge extends control to native applications and operating system input.

• Verify page appearance after deployment
• Reproduce blank pages and disappearing content
• Inspect console errors and network traffic

Crawl websites through Firecrawl to collect URLs, page content, metadata, and links. Extract content that appears through JavaScript, map site structure, and gather material for broken link checks and broader SEO analysis.

• Mapping site URLs and comparing them with an XML sitemap
• Collecting content from a blog or product section
• Extracting content rendered through JavaScript

Create or review llms.txt, a Markdown guide to a website for AI systems. Discover important pages, write factual summaries, check links and formatting, and produce a ready-to-save file or an improvement report.

• Create llms.txt from a website's public pages
• Check an existing file's format and links
• Find important pages missing from the guide

Collect product listings from Walmart category pages, including prices, ratings, availability, and seller details. Use filtered category URLs and browse multiple pages to gather structured data for product comparisons and category research.

• Collecting products from a filtered category
• Comparing prices and ratings within a category
• Checking availability and seller information

Web2 recon pipeline — subdomain enumeration (subfinder, Chaos API, assetfinder), live host discovery (dnsx, httpx), URL crawling (katana, waybackurls, gau), directory fuzzing (ffuf), JS analysis (LinkFinder, SecretFinder), continuous monitoring (new subdomain alerts, JS change detection, GitHub commit watch). Use when starting recon on any web2 target or when asked about asset discovery, subdomain enum, or attack surface mapping.

The English summary is being prepared. Showing the original description.

Web2 recon pipeline — subdomain enumeration (subfinder, Chaos API, assetfinder), live host discovery (dnsx, httpx), URL crawling (katana, waybackurls, gau), directory fuzzing (ffuf), JS analysis (LinkFinder, SecretFinder), continuous monitoring (new subdomain alerts, JS change detection, GitHub commit watch). Use when starting recon on any web2 target or when asked about asset discovery, subdomain enum, or attack surface mapping.

The English summary is being prepared. Showing the original description.

AI-powered web search, crawling, extraction, and deep research

The English summary is being prepared. Showing the original description.

Web scraping, crawling, and AI-powered answer extraction at scale

The English summary is being prepared. Showing the original description.

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.

The English summary is being prepared. Showing the original description.

Tests a deployed, activated Power Pages site at runtime using browser-based navigation, page crawling, and API request verification via Playwright. Use when the user wants to test, verify, or smoke-test their deployed site.

The English summary is being prepared. Showing the original description.

Use and read this skill immediately if the user request is in any way related to SEO or a site's organic search or AI search presence. That includes site audits, rankings, keyword research, competitors, backlinks, click or traffic changes, indexing problems, crawling, redirects, sitemaps, metadata, structured data, Core Web Vitals, internal links, content opportunities, programmatic SEO, local search, Search Console or Google Analytics questions, Google update impact, llms.txt, AI search visibility in ChatGPT, Claude, Perplexity, or Google AI Overviews, and client SEO reporting. Routes to evidence-backed local reports through the SEO CLI and MCP server.

The English summary is being prepared. Showing the original description.