Skip to content
cccskills
FreePublished on GitHub

rseng-big-data-processing

Covers processing research data that outgrows one machine's memory: out-of-core and chunked computation, Dask for scaling the scientific Python stack, Spark for distributed tabular pipelines, lazy evaluation, partitioning strategies, idempotent and restartable batch jobs, and knowing when NOT to distribute. Use when datasets no longer fit in memory, when the user mentions Dask, Spark, out-of-core or larger-than-memory data, when a pandas/NumPy workflow hits memory limits, or when designing batch pipelines over many files. (Cluster job submission and job arrays are rseng-hpc-computing; pipeline orchestration engines are rseng-workflows; profile first with rseng-performance-profiling.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24 stars(GitHub 24 + cccskills 0)

Run score Not tested

See how to install

Files included (2)

  • SKILL.md5.2 KB
  • references.md448 B

SKILL.md (original)

Review the instructions the agent will receive before you install.

Big data processing for research

"Big" starts where the current tool breaks: the dataset that no longer fits in RAM, the loop over files that no longer finishes overnight. The escalation path matters more than any framework - each step up costs complexity, debuggability and reproducibility, so take the smallest step that works.

The escalation ladder

  1. Optimize in place first: columnar formats with predicate pushdown (Parquet - rseng-scientific-file-formats), dtype downcasting, reading only needed columns; profile before scaling (rseng-performance-profiling) - many "big data" problems are memory-layout problems.
  2. Out-of-core on one machine: chunked iteration (read-process- write per chunk), memory-mapped arrays, or Dask's lazy collections on a single node. One machine with streaming processing handles far more than intuition suggests, with none of the distributed complexity.
  3. Embarrassingly parallel batch: independent per-file/per-chunk jobs as cluster job arrays (rseng-hpc-computing) or a workflow engine (rseng-workflows) - the RIGHT answer for most research sweeps, and simpler than any framework.
  4. Distributed frameworks: Dask (scales NumPy/pandas/xarray idioms; native in the Pangeo geoscience stack) or Spark (SQL-flavored tabular pipelines, industry-standard cluster tooling) when computation genuinely needs cross-partition coordination: shuffles, joins, global aggregations over larger-than-node data.

Skipping straight to step 4 is the classic mistake: a distributed job that could have been a job array is slower to build, harder to debug and harder to reproduce.

Patterns that make batch processing trustworthy

  • Idempotent tasks: running a task twice yields the same result - write to output paths derived from inputs and parameters, never append blindly.
  • Restartable pipelines: skip work whose outputs already exist (checkpointing at the task level), so a failure at file 90,000 costs minutes, not the weekend. Workflow engines give this for free (rseng-workflows).
  • Fail loudly per item, not globally: quarantine failing inputs with logged reasons and continue; a summary of 37 failures beats a crash at the first.
  • Validate at the boundaries: schema/sanity checks on ingest and before final aggregation - silent corruption scales with the data (rseng-data-management).
  • Deterministic partitioning and seeds where randomness exists, so reruns are comparable.

Framework-specific footguns

  • Lazy evaluation (Dask, Spark) means errors surface at compute time, far from their cause: materialize small samples early while developing; keep transformations testable on in-memory subsets (rseng-testing) - the same code path at toy scale is the unit test.
  • Partition sizing dominates performance: too many tiny partitions drown in scheduling overhead, too few lose parallelism; target the framework's recommended per-partition sizes and re-partition after heavy filters.
  • Shuffles (joins, groupbys across partitions) are the expensive operations - restructure to avoid them where possible, and broadcast small tables instead of joining large-to-large.
  • Cluster resources: match worker memory to partition size, and on shared clusters run the framework's scheduler inside the allocation (rseng-hpc-computing) rather than assuming the machine.
  • Record framework and cluster configuration with results - distributed runs are part of the provenance (rseng-reproducible-environments, rseng-ai-declaration for AI-assisted pipeline work).

Working with this skill

This skill is source-independent: its authority is the framework documentation and community practice linked below.

Learn more (verified):

<!-- related-skills:begin -->

Related skills

Check whether any of these applies before moving on:

  • rseng-green-computing - distributed runs carry energy cost
  • rseng-hpc-computing - job arrays and cluster allocations
  • rseng-performance-profiling - profile before scaling out
  • rseng-scientific-file-formats - Parquet and chunked stores enable it
  • rseng-testing - test transforms on in-memory subsets
  • rseng-workflows - restartable pipelines via engines
<!-- related-skills:end -->

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

Covers operating AI coding agents securely: auditing whether the agent runs sandboxed, permission and dangerous-command configuration, containerized environments, secrets kept out of context, network egress and token privileges, prompt-injection risk, and the privacy of agent use itself. Use PROACTIVELY when permission gating is broadly disabled (skip-permissions modes), when secrets are visible to the agent's shell, or when the agent processes untrusted code, issues or web content; when a project holds personal or confidential data while a cloud agent is active (what leaves the machine, retention, training use), or when the user asks how to run coding agents safely or privately. (Supply-chain security: rseng-security; disclosing agent contributions: rseng-ai-declaration.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers declaring AI involvement with the AI Declaration Format (https://ai-declaration.org): creating and maintaining an aidecl.yaml that records which AI tools and agents were used, what each contributed, when and how much, plus the README disclosure footnote, JSON/JSON-LD export and CI validation. Machine-readable disclosure is the practice; this format is one implementation. Use PROACTIVELY when an AI assistant or agent creates or modifies a project's content - generate aidecl.yaml if missing, update it after AI-assisted changes. Also use when the user asks about AI transparency, provenance, disclosure or EU AI Act disclosure, wants to state that NO AI was used, or mentions aidecl. For concealment requests see rseng-honesty; for the human review duty see rseng-human-verification.

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers long-term archiving of research software and data: Software Heritage save requests and SWHID persistent identifiers, Zenodo deposits with versioned DOIs and forge integration, choosing domain and institutional repositories, deciding what to archive (code, data, environments, documentation) and when, and archiving at project retirement. Use when the user wants software or data preserved beyond the life of a forge account, grant or lab, mentions Software Heritage, SWHIDs, Zenodo deposits or archiving, retires or hands over a project, prepares artifacts that must stay resolvable for a paper, or when funder policy requires long-term preservation. (Cutting the release itself is rseng-publishing-releasing; day-to-day data storage and repository choice is rseng-data-management.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers continuous integration and delivery for research software: CI/CD concepts, automating builds and tests with GitHub Actions and GitLab CI/CD, and wiring an organization's GitLab CI infrastructure to a GitHub-hosted project. Use when the user asks to set up CI, write a pipeline or workflow file (.github/workflows, .gitlab-ci.yml), add automated builds/tests on push or pull request, publish a package or deploy docs when a tag or release is cut, cache dependencies to speed up CI runs, choose between GitHub Actions and GitLab CI, use self-hosted or GPU runners, mirror a repo, or report external CI status back to GitHub. For what to test and shrinking a test matrix see rseng-testing; for linter and pre-commit policy see rseng-code-quality.

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers verifying that every citation is real, correct and current: checking references in manuscripts, READMEs, references files and code metadata against Crossref and OpenAlex, screening cited DOIs against the Retraction Watch database, catching fabricated or mis-attributed citations (a documented AI failure mode), and keeping bibliographies and CITATION.cff files resolvable. Use PROACTIVELY before any bibliography, reference list or citation metadata is finalized or published, whenever the agent itself has produced citations, and when the user asks to check references, mentions broken DOIs, retracted papers or citation verification, or prepares a manuscript, README or software paper with references. (Making your own software citable is rseng-citation-metadata.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers making research software citable and contributors credited: writing CITATION.cff, describing software with CodeMeta (codemeta.json), minting DOIs and ORCIDs, and tracking contributors of every kind. Use when the user asks how to make software citable, add CITATION.cff or codemeta.json, obtain a DOI, ensure contributors get credit, or mentions CFF, CodeMeta, ORCID, CRediT or persistent identifiers. Use PROACTIVELY when generated code draws on a publication, website or existing code (credit it at the code site and in the references), at release preparation, and when citation files are edited. (Verifying references you cite: rseng-citation-hygiene; versioning schemes and the DOI-minting release: rseng-publishing-releasing.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

See all skills by fdiblen

Report a problem with this skill