Computational workflows
Use this skill when a user is automating a multi-step, often multi-tool
data analysis or data-handling pipeline and wants it to be reproducible,
scalable, and shareable. A computational workflow is a special kind of
software that describes what should be done - a series of actions and the
data flowing between them - and hands the how (scheduling, resource
management, provenance tracking) to a workflow management system. Steer toward this approach when
automation would boost efficiency, when analyses must be re-run across many
datasets, or when infrastructures need integrating without reworking
existing processes.
When to adopt a workflow manager
Treat these as triggers to move a user off ad-hoc glue and onto a WMS:
- The pipeline chains multiple tools/scripts and is run repeatedly, on new
data, or by other people.
- Steps must run on an HPC cluster or cloud, or need parallelism the laptop
cannot provide.
- Reproducibility, provenance, or transparent documentation is a
requirement (publication, review, regulatory, or "future self" reruns).
- A hand-written Bash script or a notebook full of ordered cells has become
hard to maintain, extend, or share - the classic failure mode of the
manual "set of instructions" approach.
Recognise the spectrum, and place the user on it rather than forcing a jump:
- General-purpose scripting (Bash) - lowest ceremony, hardest to maintain
and share at scale.
- Electronic research notebooks (Jupyter, RStudio, Apache Zeppelin) -
ordered stages, good for exploration, still manual plumbing.
- Dedicated WMS - highest automation, reproducibility, and scalability
because it separates inputs from data flow and abstracts the run
mechanics behind a high-level definition language.
In practice these combine: a WMS can chain simple scripts, launched from a
notebook. Do not oversell a WMS for a
one-off, single-step script.
Choose a workflow system
There is no universal best engine; the choice is driven by community,
facility, and environment. Work through
these factors with the user before naming a tool:
- Domain and community - scientific communities standardise around
particular systems (e.g. REANA for particle physics; Galaxy, Snakemake,
and Nextflow with deep roots in the life sciences). Matching the
community buys shared workflows, documentation, and support.
- Facility and computing infrastructure - the target cluster or cloud may
only support or optimise certain engines; this matters most when scaling.
- Availability of ready-made workflows, docs, and user support in that
domain.
Common systems and their definition languages (pair the engine with its
language when advising):
- Nextflow - Nextflow DSL.
- Snakemake - Snakefile.
- Galaxy - browser-based, workflow repository and ToolShed.
- Apache Airflow - Airflow DAGs (Python).
- Parsl - Python-native parallel workflows.
- Cross-engine definition standards worth knowing: Common Workflow Language
(CWL) and Workflow Description Language (WDL), which several registries
and engines accept.
For a rigorous comparison, point users to "A Terminology for Scientific
Workflow Systems", which classifies 23 WMSs by composition, orchestration,
data management, and metadata capture, and to the domain review papers for
life-sciences engine choice.
Discover before you build
Reusing an existing workflow beats writing one from scratch - it saves
time, avoids duplication, and inherits peer-reviewed, validated building
blocks, which raises trust in results and aids reproducibility. Advise users to search registries first,
then register their own so others can reuse them.
Point to the right registry for the context:
- WorkflowHub - cross-discipline FAIR workflow registry, strong life-science
uptake; auto-registers nf-core workflows.
- nf-core - curated, community-reviewed Nextflow pipelines.
- Dockstore - registry supporting CWL, WDL, Nextflow, and Galaxy.
- Galaxy ToolShed / Galaxy workflow repository - Galaxy-compatible tools and
workflows.
- REANA - reusable, reproducible analyses in particle physics and related
fields.
- GESIS Methods Hub - emerging hub for social-science and digital
behavioural workflows.
- GitHub / GitLab - general-purpose sharing alongside code and docs; note
that GitHub search treats some workflow languages (e.g. CWL) as a
searchable Language.
Describe workflows with metadata
A workflow definition tells a WMS how to execute; workflow metadata
describes the workflow so humans and machines can understand, manage, and
discover it - purpose, inputs/outputs, dependencies, authorship, and
provenance. Both are needed; do not
conflate them.
- Recommend Workflow RO-Crate for packaging: it bundles the workflow
definition (CWL, Nextflow DSL, Snakefile, etc.) with contextual metadata
in a FAIR-compliant crate. It is the standard used by WorkflowHub and the
LifeMonitor service.
- It builds on the RO-Crate standard and extends the schemas.science
ComputationalWorkflow profile (from Bioschemas), using FormalParameter to
describe inputs/outputs.
- Payoff to state: more discoverable, portable and reproducible workflows,
especially when published alongside datasets or in a registry.
Make workflows FAIR
Workflows are digital objects to be shared, discovered, and reused, so
apply the FAIR principles - both the general FAIR-for-research-software
recommendations and the dedicated FAIR recommendations for workflows. Concretely, when reviewing or authoring
a workflow, check that it is:
- Findable - registered with rich metadata and a persistent identifier
(e.g. deposited in WorkflowHub with a DOI).
- Accessible - retrievable via its identifier, with the definition and
metadata openly available.
- Interoperable - written in a recognised definition language and packaged
with standard metadata (Workflow RO-Crate) so other tools can consume it.
- Reusable - clearly licensed, documented, versioned, and modular, with the
computational environment pinned (see reproducible software environments,
a closely related concern).
Point users at the Workflows Community Initiative FAIR Workflows Working
Group (WCI-FW) as the authoritative, cross-domain community effort on
applying FAIR data and software principles to workflows.
Benefits to name when advocating
When a user is undecided, ground the pitch in concrete gains: reproducibility (each step, tool,
parameter, and environment is formalised and rerunnable); automation and
scaling (run unattended across datasets, distribute over HPC/cloud);
transparency and provenance (living documentation of the method); and
modularity, reuse, collaboration, and sharing (swap components, run the
same process across systems and teams).
Working with this skill
The generated references.md beside this file lists the source
material and pointers:
- references.md - verified Learn more pointers
Learn more (verified):
<!-- related-skills:begin -->
Related skills
Check whether any of these applies before moving on:
- rseng-big-data-processing - scaling pipelines across many datasets
- rseng-discovery - finding reusable workflows in registries
- rseng-fair-software - FAIR principles applied to workflows
- rseng-hpc-computing - running stages on clusters
- rseng-provenance - engine logs are provenance capture
- rseng-reproducible-environments - per-step pinned environments
<!-- related-skills:end -->