32 skills · sorted by relevance
Use Dask to process tables, numerical arrays, and file collections in pieces across CPU cores or machines. Helps choose data structures, size memory-friendly chunks, coordinate dependent tasks, and diagnose slow or memory-heavy workflows.
Use this repo skill for Dask, the Python parallel computing library, when working with lazy task graphs, schedulers, Dask Array, Dask DataFrame, Dask Bag/bytes IO, configuration, diagnostics, CLI usage, or contributor validation.
The English summary is being prepared. Showing the original description.
Work with large scientific arrays by storing data in smaller blocks and reading selected portions. Guides compression, cloud storage, format migration, and NumPy, Dask, or Xarray integration while preserving dimensions, units, and other metadata.
Apply dask in reproducible local data workflows with version-aware APIs, explicit assumptions, and validation. Use when the user chooses dask or its strengths fit the task.
The English summary is being prepared. Showing the original description.
Specific optimization strategies for Python scripts working with larger-than-memory datasets via Dask.
The English summary is being prepared. Showing the original description.
Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV/Parquet I/O, nullable semantics, and multi-GPU DataFrame workloads.
The English summary is being prepared. Showing the original description.
Rank possible gene regulatory relationships from bulk or single-cell expression data using GRNBoost2 or GENIE3. The skill guides input preparation, compatible execution, method comparison, and stability checks to help prioritize candidates for further biological investigation.
Prepare training datasets for large language models with GPU-assisted cleaning. Covers duplicate removal, quality filtering, and personal information redaction, with workflows for text, images, video, and audio using NeMo Curator.
Prepare training datasets for large language models with GPU-assisted cleaning. Covers duplicate removal, quality filtering, and personal information redaction, with workflows for text, images, video, and audio using NeMo Curator.
Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration, NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.
The English summary is being prepared. Showing the original description.
Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.
The English summary is being prepared. Showing the original description.
This skill should be used at the start of any computationally intensive scientific task to detect and report available system resources (CPU cores, GPUs, memory, disk space). It creates a JSON file with resource information and strategic recommendations that inform computational approach decisions such as whether to use parallel processing (joblib, multiprocessing), out-of-core computing (Dask, Zarr), GPU acceleration (PyTorch, JAX), or memory-efficient strategies. Use this skill before running analyses, training models, processing large datasets, or any task where resource constraints matter.
The English summary is being prepared. Showing the original description.
Parallel/distributed computing. Scale pandas/NumPy beyond memory, parallel DataFrames/Arrays, multi-file processing, task graphs, for larger-than-RAM datasets and parallel workflows.
The English summary is being prepared. Showing the original description.
Stores and operates on sparse expression matrices for single-cell and large bulk RNA-seq, covering dgCMatrix/dgRMatrix/dgTMatrix when-each-is-fast, the dgCMatrix (CSC, R) <-> CSR (Python) implicit transpose, AnnData (cells-rows) <-> SingleCellExperiment (cells-cols) orientation flip, HDF5/h5ad vs Zarr cloud-native shift, HDF5SummarizedExperiment + DelayedArray for out-of-memory bulk, scanpy backed mode for large h5ad, the ~10-15% density crossover where dense beats sparse, 10X format proliferation (MTX vs CellRanger H5 vs h5ad), the dense-conversion memory blow-up, and Dask + Zarr for consortium-scale matrices. Use when choosing sparse format, working with single-cell-sized matrices, importing/exporting 10X, debugging R/Python interop transposes, processing matrices too large for RAM, or building cloud-native pipelines.
The English summary is being prepared. Showing the original description.
Parallel and distributed computing for large-scale data processing
The English summary is being prepared. Showing the original description.
This skill should be used at the start of any computationally intensive scientific task to detect and report available system resources (CPU cores, GPUs, memory, disk space). It creates a JSON file with resource information and strategic recommendations that inform computational approach decisions such as whether to use parallel processing (joblib, multiprocessing), out-of-core computing (Dask, Zarr), GPU acceleration (PyTorch, JAX), or memory-efficient strategies. Use this skill before running analyses, training models, processing large datasets, or any task where resource constraints matter.
The English summary is being prepared. Showing the original description.
Fast in-memory DataFrame with lazy evaluation, parallel execution, Arrow backend. Use for tabular data in RAM (1–100 GB) when pandas is too slow. Expression API: select, filter, group_by, joins, pivots, window. Lazy mode enables predicate/projection pushdown. Reads CSV, Parquet, JSON, Excel, DBs, cloud. Larger-than-RAM: Dask; GPU: cuDF.
The English summary is being prepared. Showing the original description.
Chunked N-D arrays with compression and cloud storage. NumPy-style indexing. Backends: local, S3, GCS, ZIP, memory. Dask/Xarray integration for parallel and labeled computation. For lineage use lamindb; for labeled arrays use xarray.
The English summary is being prepared. Showing the original description.
Out-of-core DataFrame for billion-row data via lazy evaluation and memory-mapped files. Use when data exceeds RAM (10 GB–TB) for fast aggregation, filtering, virtual columns, and visualization without loading. Supports HDF5, Arrow, Parquet, CSV with cloud (S3, GCS, Azure). Built-in ML transformers (scaling, PCA, K-means). In-memory: polars; distributed: dask.
The English summary is being prepared. Showing the original description.
Parallel/distributed computing for larger-than-RAM data. Components: DataFrames (parallel pandas), Arrays (parallel NumPy), Bags, Futures, Schedulers. Scales laptop to HPC cluster. For single-machine speed use polars; for out-of-core without cluster use vaex.
The English summary is being prepared. Showing the original description.
GRN inference from expression via GRNBoost2 (gradient boosting) or GENIE3 (Random Forest). Load matrix, filter by TFs, infer TF-target-importance links, save network. Dask-parallelized to single-cell scale. Core SCENIC component.
The English summary is being prepared. Showing the original description.
cuML operating skill for GPU-accelerated classical ML, cuml.accel, Dask multi-GPU workflows, data utilities, and native source-build guidance.
The English summary is being prepared. Showing the original description.
Distributed computing for larger-than-RAM pandas/NumPy workflows. Use when you need to scale existing pandas/NumPy code beyond memory or across clusters. Best for parallel file processing, distributed ML, integration with existing pandas code. For out-of-core analytics on single machine use vaex; for in-memory speed use polars.
The English summary is being prepared. Showing the original description.
Stores and operates on sparse expression matrices for single-cell and large bulk RNA-seq, covering dgCMatrix/dgRMatrix/dgTMatrix when-each-is-fast, the dgCMatrix (CSC, R) <-> CSR (Python) implicit transpose, AnnData (cells-rows) <-> SingleCellExperiment (cells-cols) orientation flip, HDF5/h5ad vs Zarr cloud-native shift, HDF5SummarizedExperiment + DelayedArray for out-of-memory bulk, scanpy backed mode for large h5ad, the ~10-15% density crossover where dense beats sparse, 10X format proliferation (MTX vs CellRanger H5 vs h5ad), the dense-conversion memory blow-up, and Dask + Zarr for consortium-scale matrices. Use when choosing sparse format, working with single-cell-sized matrices, importing/exporting 10X, debugging R/Python interop transposes, processing matrices too large for RAM, or building cloud-native pipelines.
The English summary is being prepared. Showing the original description.