9 skills · sorted by relevance
Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.
The English summary is being prepared. Showing the original description.
Train and deploy neural networks in distributed E2B sandboxes with Flow Nexus
The English summary is being prepared. Showing the original description.
Official NVIDIA-authored guidance for PhysicsNeMo ShardTensor domain parallelism — integrate domain parallelism into training/inference scripts (new or existing) with DDP or FSDP2, write and register shard patches to enable new layers/ops, and bootstrap multi-GPU correctness tests. Use when working with ShardTensor, scatter_tensor, domain parallelism, sequence/spatial sharding, ring attention, DeviceMesh + DDP/FSDP2 hybrid parallelism, or physicsnemo.domain_parallel. Do NOT use for generic PyTorch DDP/FSDP setup without domain parallelism, picking a PhysicsNeMo model or example (use physicsnemo-discover), or non-distributed training questions.
The English summary is being prepared. Showing the original description.
Train and deploy neural networks in distributed E2B sandboxes with Flow Nexus
The English summary is being prepared. Showing the original description.
Train and deploy neural networks in distributed E2B sandboxes with Flow Nexus
The English summary is being prepared. Showing the original description.
Use when training models across multiple GPUs or nodes, handling large models that don't fit in memory, or optimizing training throughput - covers DDP, FSDP, DeepSpeed ZeRO, model/data parallelism, and gradient checkpointingUse when ", " mentioned.
The English summary is being prepared. Showing the original description.
Train and deploy neural networks in distributed E2B sandboxes with Flow Nexus
The English summary is being prepared. Showing the original description.
Guides multi-GPU pre-training: DDP, FSDP2, ZeRO, tensor/pipeline/expert parallelism, fp8/Muon. Use when scaling a run, training MoE, or reproducing GPT-2 on rented GPUs.
The English summary is being prepared. Showing the original description.
Train and deploy neural networks in distributed E2B sandboxes with Flow Nexus
The English summary is being prepared. Showing the original description.