本文へ移動
cccskills
無料GitHub で公開

pyfixest-grid-sharding

Diagnose and fix slow pyfixest regression GRIDS (many feols/fepois calls run sequentially) that stay slow despite demeaner_backend="cupy64" and an idle GPU. Use when: (1) a script looping dozens of pf.feols models on a 100k+ row panel takes ~1 min/model, (2) process inspection shows ~1-1.5 cores busy and nvidia-smi shows ~0% GPU utilization with a resident cupy context, (3) planning any worker prompt that will run a model grid (robustness variants x FE structures x domains). Root cause: per-model CPU-side single-threaded fixed costs (formulaic model-matrix build, interaction construction, singleton detection, cluster vcov) dominate wall time; GPU demeaning is a small slice. Fix: shard the model grid across OS processes and/or use pyfixest multiple-estimation syntax; mandate this IN THE WORKER PROMPT.

インストール方法を見る

含まれるファイル(2)

  • SKILL.md4.6 KB
  • README.md5.1 KB

SKILL.md(原文)

インストールする前に、エージェントに与えられる指示の中身を確認できます。

pyfixest Grid Sharding

Problem

A regression grid (e.g. 2 measures x 3 FE structures x pooled+per-domain x 3 label variants ~ 70 models) on a 327k-row panel with high-cardinality director FE ran ~55 s/model sequentially — ~65 min wall — on an RTX 5080 machine with demeaner_backend="cupy64" on every call. The GPU was NOT the bottleneck.

Context / Trigger Conditions

  • Measured signature (verified 2026-07-21, H5 seat-loss rerun): job process at ~1.4 cores CPU (37.7 CPU-min in 27 wall-min), nvidia-smi 0% utilization with ~4 GB resident (cupy context loaded, idle), one pyfixest singleton warning per completed model ticking by in the log.
  • Any orchestration prompt that asks a worker to "rerun every headline cell under variants A/B/C" without specifying execution structure.

Solution

  1. Diagnose before blaming the GPU: check process CPU-minutes vs wall-clock (~1 core => serial CPU-bound) and GPU utilization (near 0% => demeaning is not the constraint). The cupy64 kwarg is still correct; it just cannot fix a CPU-dominated pipeline.
  2. Shard the GRID, not the data: split the model list across N OS processes (--shard i --nshards N over the model index, one output part-file each, merge step at the end), N ~ cores-4. Models are independent — this is the Execution Style process-sharding pattern applied to regressions.
  3. Amortize fixed costs inside a shard: build the panel/interactions ONCE per variant and reuse; where specs share RHS/FE, use pyfixest multiple- estimation syntax (multiple depvars / sw()/csw() stepwise) so one model matrix serves several reported cells.
  4. Orchestrator rule: put the sharding mandate IN the worker prompt for any grid larger than ~10 models. Workers default to sequential loops otherwise.
  5. Mid-flight call: if a sequential grid is already >1/3 done with no per-model checkpoint, let it finish — restart+shard usually nets slower. Grids launched fresh should checkpoint per model (append-only part file) so this trade-off never binds again.

Verification

Sharded reruns of the same grid should show near-linear speedup up to memory/RAM limits; per-model results must be byte-identical to the sequential run (same seeds not needed — feols is deterministic).

Measured GPU-saturation verdict (2026-07-21 escalation experiment)

A controlled escalation loop (same 327k-row seat-loss grid, N concurrent OS shard processes, nvidia-smi sampled every 2 s, RTX 5080) settled the question empirically: mean GPU utilization was 0.7% at N=4, 0.6% at N=8, and ~1% at N=12 (peaks 2-5%), with total VRAM flat around 4 GB. GPU saturation is UNATTAINABLE for pyfixest cupy64 grids — the demeaning kernel is a brief burst inside a CPU-bound per-model pipeline — so the correct objective is CPU-core saturation via process shards, with cupy64 kept on per project rules. Two further measured costs: (1) kill-and-escalate restarting loses in-flight fits (throughput FELL from 1.29 to 0.64 fits/min when escalating 4->8 mid-run) — pick N once from cores and RAM, do not escalate live; (2) each shard holds the panel in RAM (~1.2 GB for a 327k-row panel; scale linearly), so cap N by free RAM before cores. Evidence: .claude-local\specialist-directors-us\ classifier_aug_2026-07-21\stageB_v2\h5_seatloss_gpu\attempts.json.

Notes

  • VRAM: N concurrent cupy64 processes each hold a context (~4 GB observed on a 327k x 40k-FE problem); on a 16 GB card cap GPU-sharing shards at ~3 or run overflow shards with the numba default (flag them per project rules).
  • See also: [pyfixest-cupy64-absorbed-regressors] (numerical differences of the cupy backend — unrelated to speed), and the global CLAUDE.md Execution Style section (process-level parallelism; GIL makes threads useless here).

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

同じリポジトリのスキル

概要と使いどころ

Build human adjudication / hand-labeling sheets from LLM-pipeline data without evidence truncation. Use when: (1) preparing a CSV/Excel sheet for a human to rule on cases an LLM classifier or rater panel judged, (2) a labeler reports "there is no information to label from" or cells look empty in Excel, (3) excerpt columns cluster at one exact length (e.g. all 1,500 chars — a hard truncation cap). Covers: full rating-basis recovery, Excel 32,767-char cell cap, multi-line CSV mangling, ruling dropdowns, companion text files.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

N-round adversarial review pipeline for empirical research output — the chain from data to LaTeX tables to a manuscript that cites them. A Claude drafter proposes minimal diffs, a deterministic mechanical battery gates every diff from a clean state with a regression gate, a Codex reviewer files check-backed critiques, and a blind judge panel decides residual disputes. Manual-invoke ONLY: trigger when the user explicitly runs /adversarial-empirical-review or names 'adversarial-empirical-review' / 'adversarial empirical review'. Do NOT auto-trigger on generic 'review my results', 'check my tables', or manuscript-editing requests. For prose-style refinement use style-emulation instead; this skill AUDITS WHETHER THE TABLES ARE CORRECT — that each number in the tables is what the analysis code computes, reproduces from the data, and is internally consistent. It is an empirical + code review: the manuscript is read only to resolve table numbering, and prose is not examined.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Before designing, training, or auditing ANY model that replicates human-annotated labels, audit the annotation protocol's INPUT — the exact document/evidence the human labelers consulted — and give the model that same input. Use when: (1) designing a classifier/LLM extractor whose target is a hand-coded label set, (2) a label-replication model shows low recall concentrated in a label subset and the diagnosis on offer is "the label's information is not in the features", (3) reviewers propose construct splits (e.g. "designation vs record-evident"), adjudication sittings, or per-domain stop rules to explain residual disagreement with gold, (4) validating an extraction pipeline against labels transcribed from a source document. Symptom of the underlying failure: elaborate theory accumulates to explain why gold is "partially unpredictable" when the model was simply never shown the document the annotators read.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Raise real concurrency in asyncio LLM batch scorers built on the OpenAI SDK (AsyncOpenAI, including OpenAI-compatible providers like DeepSeek). Use when: (1) raising an asyncio.Semaphore above ~100 produces no throughput gain, (2) a batch pipeline saturates near 100 in-flight requests despite a larger semaphore, (3) planning a high-concurrency campaign against a provider with no hard rate limit (DeepSeek v4-flash tolerates 2000+ in flight). Root cause: AsyncOpenAI's default httpx pool caps max_connections at 100, silently bottlenecking any larger semaphore — you must pass a custom http_client with httpx.Limits sized to the semaphore.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Place pre-screened literature citations into a LaTeX or Word manuscript, or restyle the citations already in one. Three modes: (1) inline placement — inline \cite{}/\citet{}/\citep{} with a compiled references.bib, for author-date journals (APA, MLA, Harvard, Chicago author-date, IEEE, Vancouver); (2) footnote placement — full formatted \footnote{} or OOXML footnotes for legal and notes styles (Bluebook, OSCOLA, Chicago, APA, McGill) with Id./supra short forms; (3) restyle — convert existing footnote citations from one style to another. This skill is manual-invoke ONLY — trigger ONLY when the user explicitly runs /cite-placement or explicitly names the "cite-placement" skill. Do NOT auto-trigger on general citation, footnote, or reference requests.

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

Download the actual PDF binary from bot-gated sites (taxpolicycenter.org, urban.org, SSRN-hosted mirrors, think-tank/publisher sites) via the Wayback Machine id_ URL form. Use when: (1) curl/WebFetch of a .pdf URL returns HTML instead of a PDF even with a browser User-Agent, (2) pypdf fails with "invalid pdf header: b'<!DOC'" or "EOF marker not found" on a freshly downloaded file, (3) Firecrawl can parse the PDF to markdown but you need the original file on disk (e.g., filing a reference copy).

日本語の概要は準備中です。原文の説明を表示しています。

kennethkhoocy/applied-micro-skills222026年9月5日 更新

kennethkhoocy のスキルをすべて見る

このスキルの問題を報告する