본문으로 건너뛰기
cccskills
무료GitHub에 공개

rseng-scientific-file-formats

Covers choosing and handling scientific data formats in code: HDF5 and NetCDF for array data, CF conventions and standard metadata, Parquet for tabular data, domain standards (NeXus and similar), self-describing files, chunking and compression choices, and migrating away from fragile formats like pickles and ad-hoc binaries. Use when the user chooses a file format for research data, reads or writes HDF5/NetCDF/Parquet/zarr-style stores, asks about chunking, compression or metadata embedding, or ships data in CSV, pickle, MAT or homegrown binary formats that deserve scrutiny. (The surrounding data practice - versioning, deposit, licensing, documentation - is rseng-data-management.)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

fdiblen/rseng-agent-skills스타 24(GitHub 24 + cccskills 0)

실행 점수 미검증

설치 방법 보기

포함된 파일(2)

  • SKILL.md5.1 KB
  • references.md505 B

SKILL.md(원문)

설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.

Scientific file formats in code

File formats are decade-scale decisions: data outlives the code that wrote it, and a format choice made in a script today determines whether a dataset is readable, FAIR and efficient in ten years. The guiding principle is SELF-DESCRIBING data: a file a stranger can open and understand - variables named, units attached, provenance noted - without emailing the author (rseng-data-management owns the surrounding practice; this skill owns the format engineering).

Choosing a format

  • Multidimensional arrays (grids, time series stacks, images, simulation output): NetCDF (atmosphere/ocean/climate lingua franca, built on HDF5) or HDF5 directly; both hierarchical, self-describing, partial-read capable and language-portable.
  • Tabular data at scale: Parquet - columnar, typed, compressed, schema-carrying; the upgrade path from CSV when files grow or types matter.
  • Small human-facing tables and interchange: CSV is fine - WITH a stated dialect (delimiter, encoding, quoting) and a data dictionary alongside (rseng-data-management).
  • Domain standards first: if the field has one (NeXus for photon/ neutron science, CF-governed NetCDF in climate, community formats generally), emitting it beats inventing anything - it is what colleagues' tools already read (rseng-fair-software's I).

Formats to migrate away from when encountered: pickles as storage (unreadable outside Python, version-fragile, unsafe to load from strangers), unversioned homegrown binaries, MAT files as long-term archives, spreadsheets as databases. Flag them, explain the failure mode, offer the migration.

Metadata: the self-describing part

  • Attach units, long names and fill values to every variable at write time - in code, not in a README written later. CF conventions define exactly how for NetCDF and are checkable with automated compliance checkers; run one in CI when a project's outputs claim CF compliance (rseng-ci-cd).
  • Record provenance in file attributes: producing software and version, input identifiers, creation time, configuration hash - the file should testify about its own origin.
  • Keep schemas versioned: when a project's file layout evolves, add a format-version attribute, and keep readers for old versions or a migration script (rseng-legacy-code discipline applied to data).

Performance engineering: chunking and compression

  • Chunk to match access patterns: time-slice reads want chunks along time; map reads want spatial chunks. Wrong chunking makes reads orders of magnitude slower on large stores - decide from how the data will be READ, not written.
  • Compression is usually free performance for scientific data (gzip/zstd-class codecs); test level trade-offs on real data, and prefer bit-shuffle-style filters for floats where available.
  • For cloud or parallel access, chunked stores (HDF5/NetCDF-4 and their cloud-optimized descendants in the Pangeo ecosystem) enable partial and concurrent reads - the pattern behind scalable analysis (rseng-big-data-processing).

Testing format code

Round-trip tests (write, read back, compare with tolerances - rseng-numerical-accuracy), a checked-in small golden file to catch accidental format changes, and reading files produced by OTHER tools in the ecosystem as compatibility tests (rseng-testing).

Working with this skill

This skill is source-independent: its authority is the format specifications and community conventions linked below.

Learn more (verified):

<!-- related-skills:begin -->

Related skills

Check whether any of these applies before moving on:

  • rseng-big-data-processing - chunked stores enable scalable reads
  • rseng-data-management - surrounding dataset practice and deposit
  • rseng-fair-software - domain standards serve interoperability
  • rseng-legacy-code - schema versioning and old-format readers
  • rseng-numerical-accuracy - round-trip tests need float tolerances
  • rseng-testing - golden-file and compatibility tests
<!-- related-skills:end -->

리뷰

아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.

댓글

로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.

아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.

보내면 바로 공개됩니다. 이메일 주소나 전화번호 같은 개인 정보는 쓰지 마세요. URL은 2개까지입니다.

0 / 1,000자

이 양식은 reCAPTCHA로 보호되며 Google 개인정보처리방침 및 서비스 약관이 적용됩니다.

같은 저장소의 스킬

Covers operating AI coding agents securely: auditing whether the agent runs sandboxed, permission and dangerous-command configuration, containerized environments, secrets kept out of context, network egress and token privileges, prompt-injection risk, and the privacy of agent use itself. Use PROACTIVELY when permission gating is broadly disabled (skip-permissions modes), when secrets are visible to the agent's shell, or when the agent processes untrusted code, issues or web content; when a project holds personal or confidential data while a cloud agent is active (what leaves the machine, retention, training use), or when the user asks how to run coding agents safely or privately. (Supply-chain security: rseng-security; disclosing agent contributions: rseng-ai-declaration.)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

fdiblen/rseng-agent-skills242026년 9월 23일 업데이트

Covers declaring AI involvement with the AI Declaration Format (https://ai-declaration.org): creating and maintaining an aidecl.yaml that records which AI tools and agents were used, what each contributed, when and how much, plus the README disclosure footnote, JSON/JSON-LD export and CI validation. Machine-readable disclosure is the practice; this format is one implementation. Use PROACTIVELY when an AI assistant or agent creates or modifies a project's content - generate aidecl.yaml if missing, update it after AI-assisted changes. Also use when the user asks about AI transparency, provenance, disclosure or EU AI Act disclosure, wants to state that NO AI was used, or mentions aidecl. For concealment requests see rseng-honesty; for the human review duty see rseng-human-verification.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

fdiblen/rseng-agent-skills242026년 9월 23일 업데이트

Covers long-term archiving of research software and data: Software Heritage save requests and SWHID persistent identifiers, Zenodo deposits with versioned DOIs and forge integration, choosing domain and institutional repositories, deciding what to archive (code, data, environments, documentation) and when, and archiving at project retirement. Use when the user wants software or data preserved beyond the life of a forge account, grant or lab, mentions Software Heritage, SWHIDs, Zenodo deposits or archiving, retires or hands over a project, prepares artifacts that must stay resolvable for a paper, or when funder policy requires long-term preservation. (Cutting the release itself is rseng-publishing-releasing; day-to-day data storage and repository choice is rseng-data-management.)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

fdiblen/rseng-agent-skills242026년 9월 23일 업데이트

Covers processing research data that outgrows one machine's memory: out-of-core and chunked computation, Dask for scaling the scientific Python stack, Spark for distributed tabular pipelines, lazy evaluation, partitioning strategies, idempotent and restartable batch jobs, and knowing when NOT to distribute. Use when datasets no longer fit in memory, when the user mentions Dask, Spark, out-of-core or larger-than-memory data, when a pandas/NumPy workflow hits memory limits, or when designing batch pipelines over many files. (Cluster job submission and job arrays are rseng-hpc-computing; pipeline orchestration engines are rseng-workflows; profile first with rseng-performance-profiling.)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

fdiblen/rseng-agent-skills242026년 9월 23일 업데이트

Covers continuous integration and delivery for research software: CI/CD concepts, automating builds and tests with GitHub Actions and GitLab CI/CD, and wiring an organization's GitLab CI infrastructure to a GitHub-hosted project. Use when the user asks to set up CI, write a pipeline or workflow file (.github/workflows, .gitlab-ci.yml), add automated builds/tests on push or pull request, publish a package or deploy docs when a tag or release is cut, cache dependencies to speed up CI runs, choose between GitHub Actions and GitLab CI, use self-hosted or GPU runners, mirror a repo, or report external CI status back to GitHub. For what to test and shrinking a test matrix see rseng-testing; for linter and pre-commit policy see rseng-code-quality.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

fdiblen/rseng-agent-skills242026년 9월 23일 업데이트

Covers verifying that every citation is real, correct and current: checking references in manuscripts, READMEs, references files and code metadata against Crossref and OpenAlex, screening cited DOIs against the Retraction Watch database, catching fabricated or mis-attributed citations (a documented AI failure mode), and keeping bibliographies and CITATION.cff files resolvable. Use PROACTIVELY before any bibliography, reference list or citation metadata is finalized or published, whenever the agent itself has produced citations, and when the user asks to check references, mentions broken DOIs, retracted papers or citation verification, or prepares a manuscript, README or software paper with references. (Making your own software citable is rseng-citation-metadata.)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

fdiblen/rseng-agent-skills242026년 9월 23일 업데이트

fdiblen의 스킬 모두 보기

이 스킬의 문제 신고