scarray

Workflows

Standard Workflow

Convert single-cell data to GDS format, load it as a GDS-backed SingleCellExperiment, and perform memory-efficient downstream analysis.

library(SCArray)
library(SingleCellExperiment)
library(scuttle)
library(scater)

# Load example SingleCellExperiment
fn <- system.file("extdata", "example.rds", package="SCArray")
sce <- readRDS(fn)

# Convert to GDS file
scConvGDS(sce, "test.gds")

# Load SingleCellExperiment from GDS
sce_gds <- scExperiment("test.gds")

# Normalize counts
sce_gds <- logNormCounts(sce_gds)

# Row and column summarization
col_mean <- colMeans(assays(sce_gds)$counts)
mvar <- scRowMeanVar(assays(sce_gds)$counts)

# Memory-efficient PCA and UMAP
sce_gds <- scRunPCA(sce_gds)
sce_gds <- runUMAP(sce_gds)

# Plot PCA
plotReducedDim(sce_gds, dimred="PCA")

Input: A SingleCellExperiment object or matrix; Output: A GDS-backed SingleCellExperiment object with normalized counts and reduced dimensions.

When to Use

• To convert large-scale single-cell datasets (matrices or SingleCellExperiment objects) to GDS format using scConvGDS.
• To load and manipulate single-cell data out-of-memory as a GDS-backed SingleCellExperiment using scExperiment.
• To perform memory-efficient row/column summarization (e.g., colMeans, rowMeans, scRowMeanVar) on GDS-backed DelayedMatrix objects.
• To run memory-efficient PCA on large-scale single-cell datasets using scRunPCA.

When NOT to Use

• When the dataset is small enough to fit comfortably in memory and standard in-memory matrices are preferred.
• For non-array-oriented genomic data (e.g., raw FASTQ or BAM files).

Data Requirements

• Input single-cell datasets as a SingleCellExperiment object or a dense/sparse matrix (e.g., dgCMatrix).
• Output GDS files (.gds) containing compressed assay data.

Key Parameters

• SCArray.verbose (TRUE/FALSE): Global option to enable displaying debug information during SCArray operations.
• dimred: The name of the reduced dimension slot to plot in plotReducedDim (e.g., "PCA", "UMAP").

Best Practices

• Use scRunPCA instead of scater::runPCA for large-scale datasets to avoid high memory usage from realizing the full matrix in memory.
• Enable verbose debugging with options(SCArray.verbose=TRUE) to track GDS-backed matrix operations.
• Keep the GDS file connection closed when not in use by calling scClose if manually opened with scOpen.

Common Pitfalls

• Attempting to run standard memory-intensive functions (like scater::runPCA) on very large GDS-backed objects, which triggers in-memory realization and defeats the out-of-memory benefit. Use scRunPCA instead.
• Forgetting that GDS-backed matrices are DelayedMatrix objects, meaning operations are delayed until explicitly realized.

Alternatives

• HDF5Array: For HDF5-backed out-of-memory data storage and manipulation.
• TileDBArray: For TileDB-backed multi-dimensional array storage.
• LoomExperiment: For Loom-backed single-cell data storage.

Citations

• Zheng et al. (GDS format and gdsfmt package).

References

• Homepage: bioconductor.org/packages/SCArray
• Vignette: https://bioconductor.org/packages/release/bioc/vignettes/SCArray/inst/doc/SCArray.html