Skip to content
cccskills
FreePublished on GitHub

pyfixest-grid-sharding

Diagnose and fix slow pyfixest regression GRIDS (many feols/fepois calls run sequentially) that stay slow despite demeaner_backend="cupy64" and an idle GPU. Use when: (1) a script looping dozens of pf.feols models on a 100k+ row panel takes ~1 min/model, (2) process inspection shows ~1-1.5 cores busy and nvidia-smi shows ~0% GPU utilization with a resident cupy context, (3) planning any worker prompt that will run a model grid (robustness variants x FE structures x domains). Root cause: per-model CPU-side single-threaded fixed costs (formulaic model-matrix build, interaction construction, singleton detection, cluster vcov) dominate wall time; GPU demeaning is a small slice. Fix: shard the model grid across OS processes and/or use pyfixest multiple-estimation syntax; mandate this IN THE WORKER PROMPT.

The English summary is being prepared. Showing the original description.

See how to install

Files included (2)

  • SKILL.md4.6 KB
  • README.md5.1 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

pyfixest Grid Sharding

Problem

A regression grid (e.g. 2 measures x 3 FE structures x pooled+per-domain x 3 label variants ~ 70 models) on a 327k-row panel with high-cardinality director FE ran ~55 s/model sequentially — ~65 min wall — on an RTX 5080 machine with demeaner_backend="cupy64" on every call. The GPU was NOT the bottleneck.

Context / Trigger Conditions

  • Measured signature (verified 2026-07-21, H5 seat-loss rerun): job process at ~1.4 cores CPU (37.7 CPU-min in 27 wall-min), nvidia-smi 0% utilization with ~4 GB resident (cupy context loaded, idle), one pyfixest singleton warning per completed model ticking by in the log.
  • Any orchestration prompt that asks a worker to "rerun every headline cell under variants A/B/C" without specifying execution structure.

Solution

  1. Diagnose before blaming the GPU: check process CPU-minutes vs wall-clock (~1 core => serial CPU-bound) and GPU utilization (near 0% => demeaning is not the constraint). The cupy64 kwarg is still correct; it just cannot fix a CPU-dominated pipeline.
  2. Shard the GRID, not the data: split the model list across N OS processes (--shard i --nshards N over the model index, one output part-file each, merge step at the end), N ~ cores-4. Models are independent — this is the Execution Style process-sharding pattern applied to regressions.
  3. Amortize fixed costs inside a shard: build the panel/interactions ONCE per variant and reuse; where specs share RHS/FE, use pyfixest multiple- estimation syntax (multiple depvars / sw()/csw() stepwise) so one model matrix serves several reported cells.
  4. Orchestrator rule: put the sharding mandate IN the worker prompt for any grid larger than ~10 models. Workers default to sequential loops otherwise.
  5. Mid-flight call: if a sequential grid is already >1/3 done with no per-model checkpoint, let it finish — restart+shard usually nets slower. Grids launched fresh should checkpoint per model (append-only part file) so this trade-off never binds again.

Verification

Sharded reruns of the same grid should show near-linear speedup up to memory/RAM limits; per-model results must be byte-identical to the sequential run (same seeds not needed — feols is deterministic).

Measured GPU-saturation verdict (2026-07-21 escalation experiment)

A controlled escalation loop (same 327k-row seat-loss grid, N concurrent OS shard processes, nvidia-smi sampled every 2 s, RTX 5080) settled the question empirically: mean GPU utilization was 0.7% at N=4, 0.6% at N=8, and ~1% at N=12 (peaks 2-5%), with total VRAM flat around 4 GB. GPU saturation is UNATTAINABLE for pyfixest cupy64 grids — the demeaning kernel is a brief burst inside a CPU-bound per-model pipeline — so the correct objective is CPU-core saturation via process shards, with cupy64 kept on per project rules. Two further measured costs: (1) kill-and-escalate restarting loses in-flight fits (throughput FELL from 1.29 to 0.64 fits/min when escalating 4->8 mid-run) — pick N once from cores and RAM, do not escalate live; (2) each shard holds the panel in RAM (~1.2 GB for a 327k-row panel; scale linearly), so cap N by free RAM before cores. Evidence: .claude-local\specialist-directors-us\ classifier_aug_2026-07-21\stageB_v2\h5_seatloss_gpu\attempts.json.

Notes

  • VRAM: N concurrent cupy64 processes each hold a context (~4 GB observed on a 327k x 40k-FE problem); on a 16 GB card cap GPU-sharing shards at ~3 or run overflow shards with the numba default (flag them per project rules).
  • See also: [pyfixest-cupy64-absorbed-regressors] (numerical differences of the cupy backend — unrelated to speed), and the global CLAUDE.md Execution Style section (process-level parallelism; GIL makes threads useless here).

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

Build human adjudication / hand-labeling sheets from LLM-pipeline data without evidence truncation. Use when: (1) preparing a CSV/Excel sheet for a human to rule on cases an LLM classifier or rater panel judged, (2) a labeler reports "there is no information to label from" or cells look empty in Excel, (3) excerpt columns cluster at one exact length (e.g. all 1,500 chars — a hard truncation cap). Covers: full rating-basis recovery, Excel 32,767-char cell cap, multi-line CSV mangling, ruling dropdowns, companion text files.

The English summary is being prepared. Showing the original description.

kennethkhoocy/applied-micro-skills20Updated Sep 5, 2026

N-round adversarial review pipeline for empirical research output — the chain from data to LaTeX tables to a manuscript that cites them. A Claude drafter proposes minimal diffs, a deterministic mechanical battery gates every diff from a clean state with a regression gate, a Codex reviewer files check-backed critiques, and a blind judge panel decides residual disputes. Manual-invoke ONLY: trigger when the user explicitly runs /adversarial-empirical-review or names 'adversarial-empirical-review' / 'adversarial empirical review'. Do NOT auto-trigger on generic 'review my results', 'check my tables', or manuscript-editing requests. For prose-style refinement use style-emulation instead; this skill AUDITS WHETHER THE TABLES ARE CORRECT — that each number in the tables is what the analysis code computes, reproduces from the data, and is internally consistent. It is an empirical + code review: the manuscript is read only to resolve table numbering, and prose is not examined.

The English summary is being prepared. Showing the original description.

kennethkhoocy/applied-micro-skills20Updated Sep 5, 2026

Before designing, training, or auditing ANY model that replicates human-annotated labels, audit the annotation protocol's INPUT — the exact document/evidence the human labelers consulted — and give the model that same input. Use when: (1) designing a classifier/LLM extractor whose target is a hand-coded label set, (2) a label-replication model shows low recall concentrated in a label subset and the diagnosis on offer is "the label's information is not in the features", (3) reviewers propose construct splits (e.g. "designation vs record-evident"), adjudication sittings, or per-domain stop rules to explain residual disagreement with gold, (4) validating an extraction pipeline against labels transcribed from a source document. Symptom of the underlying failure: elaborate theory accumulates to explain why gold is "partially unpredictable" when the model was simply never shown the document the annotators read.

The English summary is being prepared. Showing the original description.

kennethkhoocy/applied-micro-skills20Updated Sep 5, 2026

Raise real concurrency in asyncio LLM batch scorers built on the OpenAI SDK (AsyncOpenAI, including OpenAI-compatible providers like DeepSeek). Use when: (1) raising an asyncio.Semaphore above ~100 produces no throughput gain, (2) a batch pipeline saturates near 100 in-flight requests despite a larger semaphore, (3) planning a high-concurrency campaign against a provider with no hard rate limit (DeepSeek v4-flash tolerates 2000+ in flight). Root cause: AsyncOpenAI's default httpx pool caps max_connections at 100, silently bottlenecking any larger semaphore — you must pass a custom http_client with httpx.Limits sized to the semaphore.

The English summary is being prepared. Showing the original description.

kennethkhoocy/applied-micro-skills20Updated Sep 5, 2026

Place pre-screened literature citations into a LaTeX or Word manuscript, or restyle the citations already in one. Three modes: (1) inline placement — inline \cite{}/\citet{}/\citep{} with a compiled references.bib, for author-date journals (APA, MLA, Harvard, Chicago author-date, IEEE, Vancouver); (2) footnote placement — full formatted \footnote{} or OOXML footnotes for legal and notes styles (Bluebook, OSCOLA, Chicago, APA, McGill) with Id./supra short forms; (3) restyle — convert existing footnote citations from one style to another. This skill is manual-invoke ONLY — trigger ONLY when the user explicitly runs /cite-placement or explicitly names the "cite-placement" skill. Do NOT auto-trigger on general citation, footnote, or reference requests.

The English summary is being prepared. Showing the original description.

kennethkhoocy/applied-micro-skills20Updated Sep 5, 2026

Download the actual PDF binary from bot-gated sites (taxpolicycenter.org, urban.org, SSRN-hosted mirrors, think-tank/publisher sites) via the Wayback Machine id_ URL form. Use when: (1) curl/WebFetch of a .pdf URL returns HTML instead of a PDF even with a browser User-Agent, (2) pypdf fails with "invalid pdf header: b'<!DOC'" or "EOF marker not found" on a freshly downloaded file, (3) Firecrawl can parse the PDF to markdown but you need the original file on disk (e.g., filing a reference copy).

The English summary is being prepared. Showing the original description.

kennethkhoocy/applied-micro-skills20Updated Sep 5, 2026

See all skills by kennethkhoocy

Report a problem with this skill