本文へ移動
cccskills
無料GitHub で公開

tavily-crawl

Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.

EthanYoQ/Skill-hub
インストール方法を見る

含まれるファイル（1）

  • SKILL.md3.9 KB

SKILL.md（原文）

インストールする前に、エージェントに与えられる指示の中身を確認できます。

tavily crawl

Crawl a website and extract content from multiple pages. Supports saving each page as a local markdown file.

Before running any command

If tvly is not found on PATH, install it first:

curl -fsSL https://cli.tavily.com/install.sh | bash && tvly login

Do not skip this step or fall back to other tools.

See tavily-cli for alternative install methods and auth options.

When to use

  • You need content from many pages on a site (e.g., all /docs/)
  • You want to download documentation for offline use
  • Step 4 in the workflow: search → extract → map → crawl → research

Quick start

# Basic crawl
tvly crawl "https://docs.example.com" --json

# Save each page as a markdown file
tvly crawl "https://docs.example.com" --output-dir ./docs/

# Deeper crawl with limits
tvly crawl "https://docs.example.com" --max-depth 2 --limit 50 --json

# Filter to specific paths
tvly crawl "https://example.com" --select-paths "/api/.*,/guides/.*" --exclude-paths "/blog/.*" --json

# Semantic focus (returns relevant chunks, not full pages)
tvly crawl "https://docs.example.com" --instructions "Find authentication docs" --chunks-per-source 3 --json

Options

OptionDescription
--max-depthLevels deep (1-5, default: 1)
--max-breadthLinks per page (default: 20)
--limitTotal pages cap (default: 50)
--instructionsNatural language guidance for semantic focus
--chunks-per-sourceChunks per page (1-5, requires --instructions)
--extract-depthbasic (default) or advanced
--formatmarkdown (default) or text
--select-pathsComma-separated regex patterns to include
--exclude-pathsComma-separated regex patterns to exclude
--select-domainsComma-separated regex for domains to include
--exclude-domainsComma-separated regex for domains to exclude
--allow-external / --no-externalInclude external links (default: allow)
--include-imagesInclude images
--timeoutMax wait (10-150 seconds)
-o, --outputSave JSON output to file
--output-dirSave each page as a .md file in directory
--jsonStructured JSON output

Crawl for context vs. data collection

For agentic use (feeding results to an LLM):

Always use --instructions + --chunks-per-source. Returns only relevant chunks instead of full pages — prevents context explosion.

tvly crawl "https://docs.example.com" --instructions "API authentication" --chunks-per-source 3 --json

For data collection (saving to files):

Use --output-dir without --chunks-per-source to get full pages as markdown files.

tvly crawl "https://docs.example.com" --max-depth 2 --output-dir ./docs/

Tips

  • Start conservative — --max-depth 1, --limit 20 — and scale up.
  • Use --select-paths to focus on the section you need.
  • Use map first to understand site structure before a full crawl.
  • Always set --limit to prevent runaway crawls.

See also

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

ログインしてレビューを書く

同じリポジトリのスキル

概要と使いどころ

aact-bulk-trials

無料

使用 AACT (Aggregate Analysis of ClinicalTrials.gov) PostgreSQL 数据仓库进行批量、历史、聚合性临床试验数据挖掘。Use this skill when the user requests bulk SQL analysis over the full clinical trials data warehouse — historical trial trends, disease landscapes, similar-design matching, or multi-year aggregations across hundreds of thousands of NCT records. 触发场景包括：AACT 查询、临床试验批量分析、PostgreSQL 试验数据、全量 NCT 检索、试验数据挖掘、历史试验分析、clinical trials data warehouse、SQL trials、bulk trial analysis、disease landscape、试验设计相似性匹配、跨年度聚合、sponsor/phase/country 多维统计。**与 clinical-trials-v2 差异**：本 skill 走批量 SQL · 离线大数据（PostgreSQL）；v2 走实时 API · 单查询。两者互补：单条 NCT 实时状态用 v2，百万级历史挖掘用本 skill。支持云端公共 PostgreSQL（aact-db.ctti-clinicaltrials.org · 零部署）和每日 dump 本地还原（高性能 · 离线）两种连接方式，自动检测优先用本地。跨平台（macOS/Linux/Windows）参数化 SQL 防注入，read-only 强制保护。

日本語の概要は準備中です。原文の説明を表示しています。

EthanYoQ/Skill-hub112026年10月5日 更新

aact-bulk-trials

無料

使用 AACT (Aggregate Analysis of ClinicalTrials.gov) PostgreSQL 数据库进行大批量临床试验历史分析与数据挖掘。触发场景包括：AACT 查询、临床试验批量分析、PostgreSQL 试验数据、全量 NCT 检索、试验数据挖掘、clinical trials data warehouse、疾病领域全景分析、设计相似试验匹配、跨年度试验趋势聚合。本 skill 通过 SQL 接口处理百万级试验记录，支持云端公共 PostgreSQL 服务（aact-db.ctti-clinicaltrials.org）和每日 dump 本地还原两种连接方式，自动检测并优先使用本地高性能模式。

日本語の概要は準備中です。原文の説明を表示しています。

EthanYoQ/Skill-hub112026年10月5日 更新

adaptive-quality-gate

無料

Use when work produces a deliverable, factual claim, dataset, code change, decision, publication, deployment, automation, or irreversible action whose failure would matter; when the user asks for a quality gate, acceptance criteria, QA, completeness, validation, audit, evidence, preflight, release readiness, or a definition of done; or before claiming completion on medium- or high-risk work. Automatically decide whether a formal gate is warranted, derive task-specific pass/fail criteria, gather evidence, and block unsupported completion. Make sure to use this skill even when the user does not say "quality gate" if consequential work needs acceptance criteria or completion evidence. Skip formal gating for trivial, reversible, low-impact requests.

日本語の概要は準備中です。原文の説明を表示しています。

EthanYoQ/Skill-hub112026年10月5日 更新

add-changeset

無料

Create a changeset file in .changeset/ that describes a user-facing change for the release notes

日本語の概要は準備中です。原文の説明を表示しています。

EthanYoQ/Skill-hub112026年10月5日 更新

add-competitor

無料

Add a new competitor to the AI Visibility Tool Directory — researches the tool, generates data, takes a screenshot, and inserts into the codebase

日本語の概要は準備中です。原文の説明を表示しています。

EthanYoQ/Skill-hub112026年10月5日 更新

add-lang

無料

Add tree-sitter language support to codegraph end-to-end — wire the grammar + extractor, write tests, then benchmark extraction quality and retrieval value on 3 popular real-world repos. Use when the user runs /add-lang <language> or asks to add/support a new language (e.g. Lua, Elixir, Zig, OCaml) in codegraph.

日本語の概要は準備中です。原文の説明を表示しています。

EthanYoQ/Skill-hub112026年10月5日 更新

EthanYoQ のスキルをすべて見る

このスキルの問題を報告する