Results for “scraping”

Browser automation powers web testing, scraping, and AI agent interactions. The difference between a flaky script and a reliable system comes down to understanding selectors, waiting strategies, and anti-detection patterns. This skill covers Playwright (recommended) and Puppeteer, with patterns for testing, scraping, and agentic browser control. Key insight: Playwright won the framework war. Unless you need Puppeteer's stealth ecosystem or are Chrome-only, Playwright is the better choice in 2025. Critical distinction: Testing automation (predictable apps you control) vs scraping/agent automation (unpredictable sites that fight back). Different problems, different solutions. Use when "playwright, puppeteer, browser automation, headless, web scraping, e2e test, end-to-end, selenium, chromium, browser test, page.click, locator, playwright, puppeteer, browser, automation, testing, scraping, headless, e2e, selenium, web-testing" mentioned.

The English summary is being prepared. Showing the original description.

Complete knowledge domain for Firecrawl v2 API - web scraping and crawling that converts websites into LLM-ready markdown or structured data. Use when: scraping websites, crawling entire sites, extracting web content, converting HTML to markdown, building web scrapers, handling dynamic JavaScript content, bypassing anti-bot protection, extracting structured data from web pages, or when encountering "content not loading", "JavaScript rendering issues", or "blocked by bot detection". Keywords: firecrawl, firecrawl api, web scraping, web crawler, scrape website, crawl website, extract content, html to markdown, site crawler, content extraction, web automation, firecrawl-py, firecrawl-js, llm ready data, structured data extraction, bot bypass, javascript rendering, scraping api, crawling api, map urls, batch scraping

The English summary is being prepared. Showing the original description.

Scrapes tweets from X (Twitter) by search query, user handle, or direct URL — returns full tweet data including text, author info, engagement metrics, media, and hashtags. Use when user mentions X, Twitter, tweet scraping, scrape tweets, get tweets, fetch tweets, twitter data, x.com data, tweet collector, tweet search, twitter search, social media scraping, twitter handle scraping, scrape user tweets, from:username, twitter keyword search, hashtag search, twitter timeline, twitter list scraping, collect twitter posts, export tweets, tweet analysis, twitter monitoring, search twitter, get twitter data, twitter api alternative, tweet dump.

The English summary is being prepared. Showing the original description.

Escalate blocked scraping campaigns with provider-neutral proxy rotation — decide between datacenter, residential, and mobile pools, integrate rotation with scrapling, configure session stickiness for stateful flows, monitor cost and health, and stay inside legal and ethical boundaries. Use as the next step after `headless-web-scraping` client-side stealth (StealthyFetcher, rate limiting, robots.txt) is insufficient and traffic is legitimate.

The English summary is being prepared. Showing the original description.

Extract web page content as Markdown from a single URL or a site crawl. When access is blocked, progressively switch from basic fetching to Chrome and Bright Data proxy services, which route requests through other IP addresses.

• Converting a single web page to Markdown
• Extracting content from JavaScript-rendered pages
• Retrieving pages blocked by bot detection

Web scraping with anti-bot bypass, content extraction, undocumented APIs and poison pill detection. Use when extracting content from websites, handling paywalls, implementing scraping cascades or processing social media. Covers requests, trafilatura, Playwright with stealth mode, yt-dlp and instaloader patterns.

The English summary is being prepared. Showing the original description.

Production-grade architecture patterns for Apify-powered applications. Use when designing scraping infrastructure, building multi-Actor pipelines, or integrating Apify into a larger system architecture. Trigger with "apify architecture", "apify best practices", "apify project structure", "scraping architecture", "apify system design".

The English summary is being prepared. Showing the original description.

Deploy Apify Actors and integrate scraping into external applications. Use when deploying an Actor to the platform, integrating Actor results into a web app (Next.js, Express), wiring webhooks, or scheduling scraping pipelines. Trigger with "deploy apify actor", "apify Vercel integration", "apify production deploy", "integrate apify results", "apify API endpoint".

The English summary is being prepared. Showing the original description.

Build a complete web scraping Actor with Crawlee and deploy to Apify. Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform. Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".

The English summary is being prepared. Showing the original description.

Assesses lawful basis for AI training data processing per EDPB April 2025 report on LLMs and general-purpose AI. Covers legitimate interest balancing tests, consent challenges for ML training, public dataset assessment, and web scraping lawfulness. Keywords: AI training data, lawful basis, EDPB LLM, legitimate interest, consent, web scraping.

The English summary is being prepared. Showing the original description.

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

The English summary is being prepared. Showing the original description.

Bright Data Web MCP gives MCP-compatible agents live web search, scraping, and optional browser-automation access through Bright Data’s web-access platform. It is built for agents that need current information, anti-bot resilience, and structured extraction without maintaining their own scraping stack.

The English summary is being prepared. Showing the original description.

Stealth headless browser with CDP server for anti-fingerprint browsing, parallel scraping, and low-memory CDP automation. This skill should be used when stealth browsing, anti-detection scraping, headless CDP automation with fingerprint protection, or tracker-free page fetching is needed. Triggers on 'stealth browse', 'anti-fingerprint', 'obscura', 'stealth CDP', 'undetectable scraping', 'headless stealth', 'tracker-free browsing'.

The English summary is being prepared. Showing the original description.

CLI tool for AI-powered web scraping, data extraction, search, and crawling via ScrapeGraph AI. Use when the user needs to scrape websites, extract structured data from URLs, convert pages to markdown, crawl multi-page sites, search the web for information, automate browser interactions (login, click, fill forms), get raw HTML, discover sitemaps, or generate JSON schemas. Triggers on tasks involving: (1) extracting data from websites, (2) web scraping or crawling, (3) converting webpages to markdown, (4) AI-powered web search with extraction, (5) browser automation, (6) generating output schemas for scraping. The CLI is just-scrape (npm package just-scrape).

The English summary is being prepared. Showing the original description.

Integrate Firecrawl browser actions into application code to retrieve content that requires clicks, form entry, or pagination. Use it for multi-step page interactions and workflows that need to preserve the browser session after scraping.

• Retrieving content revealed by clicks
• Working with search forms and filters
• Collecting results across multiple pages

Build web search, page content retrieval, structured data extraction, and browser interactions into applications with Firecrawl. The skill guides endpoint selection, credentials, SDK or REST integration, and verification with a real request.

• Adding web search to an application
• Extracting structured data from known URLs
• Building website clicks and form filling into a workflow

Automate forms, clicks, screenshots, and website checks in a browser where you are already signed in. A separately launched browser also supports scraping and network capture, while login or verification steps can be handed to a person.

• Fill forms on websites where you are already signed in
• Capture screenshots and check website behavior
• Record network traffic in a separately launched browser

Collect product details, prices, stock information, and customer reviews from supported online stores using Apify. Compare competitors, examine recurring quality complaints, discover sellers through Google Shopping, and export results as CSV or JSON.

• Monitor prices and compare competing products
• Analyze reviews for recurring quality issues
• Find sellers across stores through Google Shopping

Build and debug Apify Actors, cloud programs for web scraping, automation, and data processing. Get guidance on language templates, input and output schemas, runtime code, local testing, and deployment, with practices for handling credentials and external data.

• Creating an Actor for web scraping
• Defining Actor inputs and structured outputs
• Debugging an Actor with local test data

Collects every tweet in an X (Twitter) conversation thread given a conversation id (root tweet id) — the focal tweet plus all replies, sub-replies, and quote chains — and returns normalized per-tweet data with text, author, engagement counts, media, hashtags, mentions, in_reply_to mapping, and cursor for pagination. Use when user mentions Twitter conversation, X conversation thread, conversation_id, thread scraper, scrape Twitter replies, all replies to a tweet, replies under a tweet, sub-replies, nested replies, thread harvester, get replies of a tweet, scrape comments on Twitter, scrape comments on X, full thread extraction, conversation export, conversation tree, reply chain, thread dump, X tweet thread, twitter thread scrape, comment scraping twitter, comment scraping x, focal tweet plus context, root tweet plus replies. Also applies to sentiment analysis on a single viral tweet, controversy mapping, harvesting community Q&A threads, capturing AMA threads, recovering long-running discussions, and any paginated bulk reply collection driven by a conversation id.

The English summary is being prepared. Showing the original description.

Extract complete product information from any e-commerce product page. Returns name, price, currency, brand, images, description, SKU/ASIN/EAN/UPC/GTIN/MPN identifiers, stock availability, rating, review count, variants, and seller. Works on Shopify, Amazon, WooCommerce, eBay, Walmart, Etsy, AliExpress, Alibaba, Target, Best Buy, Rakuten, Magento, BigCommerce, PrestaShop, and any public e-commerce site. Accepts product URL, keyword, or product identifier (SKU/ASIN/EAN/UPC). Use when: scrape product page, get product details, extract price and availability, product info extraction, check product data, product detail scraping, get product from URL, keyword product search, ASIN lookup, EAN search, UPC lookup, price check, product research, compare products, monitor product price, get product images, product brand and description, ecommerce data extraction, product catalog scraping, product page scraper, get item price, fetch product info.

The English summary is being prepared. Showing the original description.

Implement Apify webhooks for Actor run notifications and event-driven pipelines. Use when setting up run completion alerts, building event-driven scraping pipelines, or configuring ad-hoc webhooks for individual runs. Trigger with "apify webhook", "apify events", "actor run notification", "apify run succeeded webhook", "apify ad-hoc webhook".

The English summary is being prepared. Showing the original description.

Web data extraction using 55+ Apify Actors for AI-driven scraping. Supports Instagram, Facebook, TikTok, YouTube, Google, and more. Auto-selects best Actor for the task. Structured output in JSON/CSV with rate limiting and ethical scraping guidelines.

The English summary is being prepared. Showing the original description.

Compatibility alias for the original ScrapeCreators social scraping skill. Use scrapecreators-api for current endpoint guidance and the task-specific Brand Growth research skills for finished analysis.

The English summary is being prepared. Showing the original description.