Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes
The English summary is being prepared. Showing the original description.
Run score Not tested
See how to installReview the instructions the agent will receive before you install.
Extract structured data from websites using BeautifulSoup and requests - turn any webpage into usable data.
| Claude Does | You Decide |
|---|---|
| Structures analysis frameworks | Strategic priorities |
| Synthesizes market data | Competitive positioning |
| Identifies opportunities | Resource allocation |
| Creates strategic options | Final strategy selection |
| Suggests implementation approaches | Execution decisions |
pip install beautifulsoup4 requests pandas click lxml
python scripts/main.py scrape https://example.com --selector "h1,h2,p"
python scripts/main.py scrape https://example.com --selector ".product-price"
python scripts/main.py links https://example.com
python scripts/main.py links https://example.com --internal-only
python scripts/main.py emails https://example.com
python scripts/main.py emails https://example.com --depth 2
python scripts/main.py structured https://example.com/article --schema article
python scripts/main.py structured https://example.com/product --schema product
python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"
# Output:
# Extracted 6 elements
# 1. Starter - $29/mo
# 2. Pro - $99/mo
# 3. Enterprise - Contact us
python scripts/main.py structured https://blog.example.com/post --schema article
# Output: article_data.json
# {
# "title": "How to Scale Your Startup",
# "author": "Jane Doe",
# "date": "2024-01-15",
# "content": "...",
# "word_count": 1523
# }
| Selector | Description | Example |
|---|---|---|
tag | Element type | h1, p, div |
.class | Class name | .price, .title |
#id | Element ID | #main-content |
tag.class | Tag with class | div.product |
tag[attr] | Has attribute | a[href] |
parent > child | Direct child | ul > li |
tag1, tag2 | Multiple | h1, h2, h3 |
category: automation
subcategory: data-extraction
dependencies: [beautifulsoup4, requests, pandas]
difficulty: intermediate
time_saved: 5+ hours/week
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Measure and optimize growth using the AARRR (Pirate Metrics) framework with stage-specific KPIs and funnel analysis
The English summary is being prepared. Showing the original description.
Calculate A/B test statistical significance. Use when: determining if test results are significant; calculating required sample size; estimating test duration; analyzing conversion experiments; making data-driven decisions
The English summary is being prepared. Showing the original description.
Assess customer account health using product usage, support sentiment, payment status, and relationship signals
The English summary is being prepared. Showing the original description.
Analyze paid advertising performance across channels and recommend budget reallocation to maximize ROAS and minimize CAC. Use when: planning quarterly ad budget allocation, diagnosing underperforming ad channels, deciding whether to scale spend on a channel, calculating marginal ROI across Google Ads, Meta, LinkedIn, or TikTok, rebalancing media mix after performance shifts, or setting up a test-and-scale framework for new channels.
The English summary is being prepared. Showing the original description.
Use when testing skills, commands, or agents for quality. Use after creating new skills, before deploying agents, or when debugging inconsistent agent behavior. Triggers on "evaluate", "test quality", "is this skill working", or QA of AI workflows.
The English summary is being prepared. Showing the original description.
Use when analyzing server logs to understand how AI crawlers (GPTBot, ClaudeBot, PerplexityBot) interact with your site. Use when optimizing content placement for LLM retrieval, diagnosing why AI search isn't citing your content, or auditing crawl patterns to find optimization gaps.
The English summary is being prepared. Showing the original description.