14 skills · sorted by relevance
Add PyTorch FSDP2 to training scripts so model state can be split across GPUs. Covers distributed initialization, mixed precision, CPU offloading, and checkpoint saving and loading for models that exceed a single GPU's memory.
Get guidance on distributed training with PyTorch FSDP, which splits model parameters across devices. The skill covers implementation, mixed precision, moving data to CPU memory, communication setup, and debugging distributed training code.
Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.
Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.
Official NVIDIA-authored guidance for PhysicsNeMo ShardTensor domain parallelism — integrate domain parallelism into training/inference scripts (new or existing) with DDP or FSDP2, write and register shard patches to enable new layers/ops, and bootstrap multi-GPU correctness tests. Use when working with ShardTensor, scatter_tensor, domain parallelism, sequence/spatial sharding, ring attention, DeviceMesh + DDP/FSDP2 hybrid parallelism, or physicsnemo.domain_parallel. Do NOT use for generic PyTorch DDP/FSDP setup without domain parallelism, picking a PhysicsNeMo model or example (use physicsnemo-discover), or non-distributed training questions.
The English summary is being prepared. Showing the original description.
Fornece treinamento distribuído de LLM nativo do PyTorch usando torchtitan com paralelismo 4D (FSDP2, TP, PP, CP). Use para fazer pré-treinamento de Llama 3.1, DeepSeek V3 ou modelos customizados em escala de 8 a 512+ GPUs com Float8, torch.compile e checkpoint distribuído.
The English summary is being prepared. Showing the original description.
Orientação especializada para treinamento com Fully Sharded Data Parallel do PyTorch (FSDP) - sharding de parâmetros, precisão mista, CPU offloading, FSDP2
The English summary is being prepared. Showing the original description.
Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP - parameter sharding, mixed precision, CPU offloading, FSDP2
The English summary is being prepared. Showing the original description.
Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
The English summary is being prepared. Showing the original description.
Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP - parameter sharding, mixed precision, CPU offloading, FSDP2
The English summary is being prepared. Showing the original description.
Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.
The English summary is being prepared. Showing the original description.
Guides multi-GPU pre-training: DDP, FSDP2, ZeRO, tensor/pipeline/expert parallelism, fp8/Muon. Use when scaling a run, training MoE, or reproducing GPT-2 on rented GPUs.
The English summary is being prepared. Showing the original description.
Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP - parameter sharding, mixed precision, CPU offloading, FSDP2
The English summary is being prepared. Showing the original description.
Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
The English summary is being prepared. Showing the original description.