本文へ移動
cccskills
無料GitHub で公開

download-gated-pdfs

Download the actual PDF binary from bot-gated sites (taxpolicycenter.org, urban.org, SSRN-hosted mirrors, think-tank/publisher sites) via the Wayback Machine id_ URL form. Use when: (1) curl/WebFetch of a .pdf URL returns HTML instead of a PDF even with a browser User-Agent, (2) pypdf fails with "invalid pdf header: b'<!DOC'" or "EOF marker not found" on a freshly downloaded file, (3) Firecrawl can parse the PDF to markdown but you need the original file on disk (e.g., filing a reference copy).

インストール方法を見る

含まれるファイル(2)

  • SKILL.md2.9 KB
  • README.md3.4 KB

SKILL.md(原文)

インストールする前に、エージェントに与えられる指示の中身を確認できます。

Download bot-gated PDFs via Wayback id_

Problem

Many think-tank and publisher sites (taxpolicycenter.org, urban.org, SSRN delivery) serve an HTML bot-challenge page instead of the PDF to non-browser clients. A browser User-Agent header does not help. The downloaded "PDF" is actually HTML.

Context / Trigger Conditions

  • curl -o file.pdf <url> succeeds but the file starts with <!DOC
  • pypdf raises invalid pdf header: b'<!DOC' or PdfStreamError: Stream has ended unexpectedly
  • Firecrawl scrape returns clean markdown for the same URL (its proxies get through), but Firecrawl does not return the binary — only parsed content

Solution

  1. Request the file through the Wayback Machine's raw-content (id_) endpoint, which serves the original archived binary without rewriting:
    curl -sL -A "Mozilla/5.0 ... Chrome/126.0 Safari/537.36" \
      "https://web.archive.org/web/<YYYY>id_/<original-pdf-url>" -o out.pdf
    
    <YYYY> is any year likely to have a snapshot (e.g. publication year); Wayback redirects to the nearest capture. The id_ suffix after the timestamp is what requests the untouched original.
  2. Verify the download with pypdf — a bot page fails immediately:
    from pypdf import PdfReader
    r = PdfReader("out.pdf"); print(len(r.pages), "pages")
    
  3. If Wayback has no capture, fall back to: another mirror found via search (Exa/Firecrawl), or Firecrawl scrape for the parsed text when the binary is not strictly needed.

Verification

PdfReader opens the file and reports a plausible page count; first-page text matches the expected title.

Example

Verified 2026-07-15: taxpolicycenter.org/sites/default/files/publication/165884/ssrn-id4797771.pdf and urban.org/sites/default/files/publication/80621/2000790-...pdf both bot-gated to direct curl (with UA), both downloaded intact via https://web.archive.org/web/2024id_/<url> and .../web/2023id_/<url> (18 and 12 pages).

Notes

  • Government data hosts (e.g. ticdata.treasury.gov) are usually NOT gated — try direct curl first; Wayback is the fallback, not the default.
  • Wayback captures can be stale for frequently-revised documents; check the snapshot date if currency matters.
  • See also: the pdf skill (parsing/extraction after download) and firecrawl:firecrawl-scrape (parsed markdown when the binary is unnecessary).

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

同じリポジトリのスキル

概要と使いどころ

Build human adjudication / hand-labeling sheets from LLM-pipeline data without evidence truncation. Use when: (1) preparing a CSV/Excel sheet for a human to rule on cases an LLM classifier or rater panel judged, (2) a labeler reports "there is no information to label from" or cells look empty in Excel, (3) excerpt columns cluster at one exact length (e.g. all 1,500 chars — a hard truncation cap). Covers: full rating-basis recovery, Excel 32,767-char cell cap, multi-line CSV mangling, ruling dropdowns, companion text files.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

N-round adversarial review pipeline for empirical research output — the chain from data to LaTeX tables to a manuscript that cites them. A Claude drafter proposes minimal diffs, a deterministic mechanical battery gates every diff from a clean state with a regression gate, a Codex reviewer files check-backed critiques, and a blind judge panel decides residual disputes. Manual-invoke ONLY: trigger when the user explicitly runs /adversarial-empirical-review or names 'adversarial-empirical-review' / 'adversarial empirical review'. Do NOT auto-trigger on generic 'review my results', 'check my tables', or manuscript-editing requests. For prose-style refinement use style-emulation instead; this skill AUDITS WHETHER THE TABLES ARE CORRECT — that each number in the tables is what the analysis code computes, reproduces from the data, and is internally consistent. It is an empirical + code review: the manuscript is read only to resolve table numbering, and prose is not examined.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Before designing, training, or auditing ANY model that replicates human-annotated labels, audit the annotation protocol's INPUT — the exact document/evidence the human labelers consulted — and give the model that same input. Use when: (1) designing a classifier/LLM extractor whose target is a hand-coded label set, (2) a label-replication model shows low recall concentrated in a label subset and the diagnosis on offer is "the label's information is not in the features", (3) reviewers propose construct splits (e.g. "designation vs record-evident"), adjudication sittings, or per-domain stop rules to explain residual disagreement with gold, (4) validating an extraction pipeline against labels transcribed from a source document. Symptom of the underlying failure: elaborate theory accumulates to explain why gold is "partially unpredictable" when the model was simply never shown the document the annotators read.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Raise real concurrency in asyncio LLM batch scorers built on the OpenAI SDK (AsyncOpenAI, including OpenAI-compatible providers like DeepSeek). Use when: (1) raising an asyncio.Semaphore above ~100 produces no throughput gain, (2) a batch pipeline saturates near 100 in-flight requests despite a larger semaphore, (3) planning a high-concurrency campaign against a provider with no hard rate limit (DeepSeek v4-flash tolerates 2000+ in flight). Root cause: AsyncOpenAI's default httpx pool caps max_connections at 100, silently bottlenecking any larger semaphore — you must pass a custom http_client with httpx.Limits sized to the semaphore.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Place pre-screened literature citations into a LaTeX or Word manuscript, or restyle the citations already in one. Three modes: (1) inline placement — inline \cite{}/\citet{}/\citep{} with a compiled references.bib, for author-date journals (APA, MLA, Harvard, Chicago author-date, IEEE, Vancouver); (2) footnote placement — full formatted \footnote{} or OOXML footnotes for legal and notes styles (Bluebook, OSCOLA, Chicago, APA, McGill) with Id./supra short forms; (3) restyle — convert existing footnote citations from one style to another. This skill is manual-invoke ONLY — trigger ONLY when the user explicitly runs /cite-placement or explicitly names the "cite-placement" skill. Do NOT auto-trigger on general citation, footnote, or reference requests.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Complete methodology for computing publication-quality cumulative abnormal returns with proper event-study test statistics, matching the robustness of Kaspereit's eventstudy2 for Stata. Covers dateline construction, event-date mapping, estimation and event windows, thin-trading adjustment, OLS with Theil prediction error correction, abnormal return computation, CAR/CAAR/AAR accumulation, boundary contamination guards, and common tests such as Patell, BMP, Kolari-Pynnonen, generalized sign, Wilcoxon, and GRANK-T. Use when the user mentions abnormal returns, event windows, market-model regressions, CARs, CAAR, AAR, eventstudy2, thin trading, trade-to-trade returns, or event-study test statistics.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

kennethkhoocy のスキルをすべて見る

このスキルの問題を報告する