Captures web sources and compiles deterministic Markdown reference bundles with metadata and checksums. Use when asked to capture URLs, research web links, bundle search results, or archive web pages into an LLM-ready reference document.
The English summary is being prepared. Showing the original description.
Review the instructions the agent will receive before you install.
Capture URLs, file lists, or search results into deterministic, verifiable Markdown reference bundles with SHA-256 provenance using web-source-bundler.
web-source-bundler requires Python 3.10+ and Playwright.
Install from GitHub repository:
pip install git+https://github.com/AndrewBrown-USDA/web-source-bundler.git
(Or if developing locally within the repository workspace: pip install -e .)
Install Playwright browser binaries:
python -m playwright install chromium
Both web-source-bundler and source-bundler CLI entrypoints are available once installed.
https://....txt with one URL per line)url or link, title, and optional snippet)--out <dir> (defaults to ./source-bundle-<timestamp>)--interactive if user requested manual selection--no-screenshot, --no-pdf, --include-raw-html--redact-pattern "<regex>" if sensitive tokens/PII are mentioned--include-links-table if an appendix of all extracted links is requestedweb-source-bundler command:
web-source-bundler urls https://example.com https://another.com --out ./bundle
web-source-bundler file urls.txt --out ./bundle
web-source-bundler search-results results.json --out ./bundle
manifest.json to verify capture status and review any reported errors.combined.md for conversational grounding and reference citation by Source ID (e.g., Source 001).combined.md.A complete bundle follows this directory structure:
source-bundle-20260918-120000/
+-- manifest.json # Provenance metadata, capture config, and error logs
+-- combined.md # Single consolidated reference file for AI prompts
+-- checksums.sha256 # SHA-256 hashes of all files in bundle
+-- sources/
+-- 001/
| +-- metadata.json # Source metadata, HTTP status, and timings
| +-- readable.md # Clean Markdown with YAML front matter
| +-- rendered.html # Rendered DOM HTML
| +-- ast.json # Block AST of extracted text
| +-- screenshot.png # Full-page screenshot (if enabled)
| +-- page.pdf # Vector PDF printout (if enabled)
| +-- raw.html # Raw HTTP body (with --include-raw-html)
| +-- response_headers.json # HTTP headers (with --include-raw-html)
+-- 002/
+-- metadata.json
+-- readable.md
combined.md)combined.md joins all sources into one document for AI prompts:
# Source Bundle
Bundle ID: source-bundle-20260918-120000
Created: 2026-09-18T12:00:00Z
Tool: web-source-bundler 0.1.0
Use this document as the reference source bundle. Cite material by source ID.
---
## Source 001: Example Domain
- Input URL: https://example.com
- Final URL: https://example.com/
- Fetched: 2026-09-18T12:00:05Z
- SHA-256 readable Markdown: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
### Content
---
source_id: 001
title: "Example Domain"
input_url: https://example.com
final_url: https://example.com/
fetched_at: 2026-09-18T12:00:05Z
http_status: 200
---
Example Domain
This domain is for use in illustrative examples in documents.
| Anti-Pattern | Why It's Bad | Fix |
|---|---|---|
| Ad-hoc unchecksummed scraping | Misses dynamic JS rendering, screenshots, and PDFs | Install and run web-source-bundler with headless Playwright |
| Silent error swallowing | Corrupts audit records and leads to hallucinated sources | Check manifest.json and report any failed URLs |
| Omitting Source ID citations | Loses verifiable traceability back to original web source | Always cite findings by Source ID (e.g. Source 001) from combined.md |
| Unstructured output dumps | Fragile for downstream LLM reasoning and indexing | Always output standardized manifest.json and combined.md |
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Distills enduring architectural decisions and design rationale from wave orchestration plans or historical git changes into structured, tracked Architecture Decision Records (ADRs). Use pre-wave to resolve active architectural constraints or post-wave to record enduring decisions in docs/decisions/.
The English summary is being prepared. Showing the original description.
Create and maintain AGENTS.md files for project onboarding and grounding. Use when initialized in a new project, when project structure changes significantly, or when creating subfolder-level AGENTS.md files for complex monorepos.
The English summary is being prepared. Showing the original description.
Audit and update an arbitrary codebase to follow the Code.mil federal open-source licensing model (MIT with 17 U.S.C. Section 105 disclaimers and INTENT.md), including CRAN/R package DESCRIPTION setup.
The English summary is being prepared. Showing the original description.
Migrates ArcPy and ArcGIS Pro workflows to explicit, testable open-source Python geospatial workflows. Use when removing ArcPy dependencies, porting geoprocessing scripts, or documenting ArcGIS behavior that will not be migrated.
The English summary is being prepared. Showing the original description.
Ultra-compressed communication mode that cuts output tokens while keeping technical accuracy. Levels: lite, full, ultra and the wenyan variants. Use for /caveman, "caveman mode", "talk like caveman", "be brief" or "less tokens".
The English summary is being prepared. Showing the original description.
Review the changes since a fixed point (commit, branch, tag, or merge-base) along two axes - Standards (does the code follow this repo's documented coding standards?) and Spec (does the code match what the originating issue/spec asked for?). Runs both reviews in parallel sub-agents and reports them side by side. Use when the user wants to review a branch, a PR, work-in-progress changes, or asks to "review since X".
The English summary is being prepared. Showing the original description.