Skip to content
cccskills
FreePublished on GitHub

rseng-hpc-computing

Covers working effectively on high-performance computing clusters: writing and debugging SLURM job scripts, choosing and requesting resources honestly, running containers with Apptainer, using module systems and EESSI software stacks, MPI basics, checkpointing, and scaling from laptop to cluster reproducibly. Use when the user mentions a cluster, supercomputer, SLURM, sbatch, MPI, Apptainer or Singularity, module load, job arrays or walltime, or when a compute workload has outgrown a single machine. (Larger-than-memory data processing with Dask or Spark is rseng-big-data-processing; measuring scaling before requesting allocations is rseng-performance-profiling.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24 stars(GitHub 24 + cccskills 0)

Run score Not tested

See how to install

Files included (2)

  • SKILL.md5.6 KB
  • references.md524 B

SKILL.md (original)

Review the instructions the agent will receive before you install.

Research computing on HPC clusters

HPC clusters are where much research code actually runs, and they invert desktop habits: you do not run programs, you request resources and submit batch jobs; you do not install software, you load modules or bring containers; the login node is a shared hallway, not a workstation. An agent's job is to translate the user's computation into this model correctly and reproducibly - and to respect that every cluster has local documentation that overrides generic advice.

Batch jobs done right (SLURM)

A job script is a shell script with resource directives:

#!/bin/bash
#SBATCH --job-name=fit-model
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --output=logs/%x-%j.out

module load Python/3.12
srun python fit_model.py --config configs/run.yaml

Practices that matter:

  • Request honestly: measure a pilot run, then set time/memory with modest headroom. Over-requesting wastes allocation and queue priority (and energy - rseng-green-computing); under-requesting kills jobs at 99%.
  • Never compute on the login node; test with a short interactive allocation (srun --pty or salloc) instead.
  • Parameter sweeps are job arrays (--array=0-99), not 100 submitted scripts; index into a config list with $SLURM_ARRAY_TASK_ID (externalized run configs pair naturally with rseng-workflows).
  • Long jobs checkpoint: clusters preempt and nodes fail; save restartable state at intervals and make the script resume from the last checkpoint.
  • Log usefully: %x-%j names, and seff JOBID after completion to compare requested vs used resources - feed that back into the next request.

Software environments on clusters

Three layers, in order of preference for reproducibility:

  1. Containers with Apptainer (formerly Singularity): the HPC-native container runtime - unprivileged, image-file based, GPU-aware. Build once (possibly from a Docker image), run identically on any cluster; pairs with rseng-reproducible-environments for the build recipe.
  2. EESSI: a shared, optimized, ready-to-use scientific software stack streamed to any cluster or laptop - when available it gives identical software everywhere without building anything.
  3. Module system (module avail / module load): the site's curated builds; record exact module versions in the job script - an unversioned module load Python is a reproducibility hole.

Never pip-install into the home directory as the primary strategy; quota, node-architecture and reproducibility problems follow.

Parallelism: pick the right kind

  • Independent runs (sweeps, bootstraps, per-file processing): job arrays or a workflow engine - no MPI needed. This covers most research workloads.
  • Single-node parallelism: threads/multiprocessing; --cpus-per-task, and pin BLAS/OpenMP thread counts to match.
  • Multi-node tightly-coupled computation: MPI (mpi4py or native), launched with srun; requires the code to be written for it. Distributed data processing frameworks are their own path (rseng-big-data-processing).

Scaling discipline: measure strong/weak scaling on small counts before requesting large allocations; doubling cores that yield 1.2x wastes everyone's allocation (rseng-performance-profiling for the measurement).

Cluster citizenship and data

  • Filesystems differ: home (small, backed up), project (shared), scratch (fast, purged) - stage data to scratch for jobs, move results back, and script the staging (rseng-data-management).
  • Transfer with rsync (resumable) or the site's data-transfer nodes; never leave the only copy of results on scratch.
  • Read the site's own docs first; when the user's cluster is known, prefer its conventions over anything generic, including this skill.

Working with this skill

This skill is source-independent: its authority is the scheduler, container and software-stack documentation linked below, plus each cluster's local documentation.

Learn more (verified):

<!-- related-skills:begin -->

Related skills

Check whether any of these applies before moving on:

  • rseng-big-data-processing - distributed data framework path
  • rseng-data-management - staging data across cluster filesystems
  • rseng-green-computing - honest requests save energy
  • rseng-performance-profiling - measure scaling before allocating
  • rseng-reproducible-environments - containers and pinned modules
  • rseng-workflows - sweeps via workflow engines
<!-- related-skills:end -->

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

Covers operating AI coding agents securely: auditing whether the agent runs sandboxed, permission and dangerous-command configuration, containerized environments, secrets kept out of context, network egress and token privileges, prompt-injection risk, and the privacy of agent use itself. Use PROACTIVELY when permission gating is broadly disabled (skip-permissions modes), when secrets are visible to the agent's shell, or when the agent processes untrusted code, issues or web content; when a project holds personal or confidential data while a cloud agent is active (what leaves the machine, retention, training use), or when the user asks how to run coding agents safely or privately. (Supply-chain security: rseng-security; disclosing agent contributions: rseng-ai-declaration.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers declaring AI involvement with the AI Declaration Format (https://ai-declaration.org): creating and maintaining an aidecl.yaml that records which AI tools and agents were used, what each contributed, when and how much, plus the README disclosure footnote, JSON/JSON-LD export and CI validation. Machine-readable disclosure is the practice; this format is one implementation. Use PROACTIVELY when an AI assistant or agent creates or modifies a project's content - generate aidecl.yaml if missing, update it after AI-assisted changes. Also use when the user asks about AI transparency, provenance, disclosure or EU AI Act disclosure, wants to state that NO AI was used, or mentions aidecl. For concealment requests see rseng-honesty; for the human review duty see rseng-human-verification.

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers long-term archiving of research software and data: Software Heritage save requests and SWHID persistent identifiers, Zenodo deposits with versioned DOIs and forge integration, choosing domain and institutional repositories, deciding what to archive (code, data, environments, documentation) and when, and archiving at project retirement. Use when the user wants software or data preserved beyond the life of a forge account, grant or lab, mentions Software Heritage, SWHIDs, Zenodo deposits or archiving, retires or hands over a project, prepares artifacts that must stay resolvable for a paper, or when funder policy requires long-term preservation. (Cutting the release itself is rseng-publishing-releasing; day-to-day data storage and repository choice is rseng-data-management.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers processing research data that outgrows one machine's memory: out-of-core and chunked computation, Dask for scaling the scientific Python stack, Spark for distributed tabular pipelines, lazy evaluation, partitioning strategies, idempotent and restartable batch jobs, and knowing when NOT to distribute. Use when datasets no longer fit in memory, when the user mentions Dask, Spark, out-of-core or larger-than-memory data, when a pandas/NumPy workflow hits memory limits, or when designing batch pipelines over many files. (Cluster job submission and job arrays are rseng-hpc-computing; pipeline orchestration engines are rseng-workflows; profile first with rseng-performance-profiling.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers continuous integration and delivery for research software: CI/CD concepts, automating builds and tests with GitHub Actions and GitLab CI/CD, and wiring an organization's GitLab CI infrastructure to a GitHub-hosted project. Use when the user asks to set up CI, write a pipeline or workflow file (.github/workflows, .gitlab-ci.yml), add automated builds/tests on push or pull request, publish a package or deploy docs when a tag or release is cut, cache dependencies to speed up CI runs, choose between GitHub Actions and GitLab CI, use self-hosted or GPU runners, mirror a repo, or report external CI status back to GitHub. For what to test and shrinking a test matrix see rseng-testing; for linter and pre-commit policy see rseng-code-quality.

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

Covers verifying that every citation is real, correct and current: checking references in manuscripts, READMEs, references files and code metadata against Crossref and OpenAlex, screening cited DOIs against the Retraction Watch database, catching fabricated or mis-attributed citations (a documented AI failure mode), and keeping bibliographies and CITATION.cff files resolvable. Use PROACTIVELY before any bibliography, reference list or citation metadata is finalized or published, whenever the agent itself has produced citations, and when the user asks to check references, mentions broken DOIs, retracted papers or citation verification, or prepares a manuscript, README or software paper with references. (Making your own software citable is rseng-citation-metadata.)

The English summary is being prepared. Showing the original description.

fdiblen/rseng-agent-skills24Updated Sep 23, 2026

See all skills by fdiblen

Report a problem with this skill