Results for “distributed training”

Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.

• Pretraining Llama 3.1 8B on eight GPUs
• Configuring multi-node training with SLURM
• Enabling Float8 training on H100 GPUs

Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.

• Pretraining Llama 3.1 8B on eight GPUs
• Configuring multi-node training with SLURM
• Enabling Float8 training on H100 GPUs

Guide large language model training with OpenRLHF using human preference feedback. Covers training method selection, distributed GPU setup, reward model training, and memory tuning, with examples for PPO, GRPO, and DPO.

• Choosing a training method for human preference alignment
• Setting up distributed training across multiple GPUs
• Training a reward model before PPO

Plan and configure large language model training with OpenRLHF to better match human preferences. Covers training method selection, distributed GPU setup, inference acceleration with vLLM, and troubleshooting memory or training stability issues.

• Choosing a method for training models on human preferences
• Setting up PPO training with a reward model
• Configuring GRPO training without a critic model

Guides LLM training with verl using rewards to improve model responses. Covers data preparation, reward functions, GRPO and PPO configuration, distributed GPU training, and troubleshooting memory shortages or unstable training.

• Train math reasoning models with GRPO and answer-based rewards
• Configure PPO training with a separate value-estimation model
• Run large-model training across multiple GPU nodes

Adapt existing PyTorch training scripts to run across multiple GPUs or machines with Accelerate. This skill covers device setup, distributed training configuration, and mixed precision, which uses lower-precision numbers to reduce memory use.

• Adapt a single-GPU training script for multiple GPUs
• Configure mixed precision to reduce training memory use
• Set up training with DeepSpeed or FSDP

Organize PyTorch training code with LightningModule and Trainer. Follow examples for validation, logging, model saving, and distributed training across multiple GPUs, with callbacks that add monitoring or stop training when progress stalls.

• Converting an existing PyTorch training loop
• Adding validation, logs, and model checkpoints
• Configuring training across multiple GPUs

Get guidance on distributed training with PyTorch FSDP, which splits model parameters across devices. The skill covers implementation, mixed precision, moving data to CPU memory, communication setup, and debugging distributed training code.

• Implement distributed training with FSDP
• Review mixed precision and CPU offloading settings
• Choose a distributed communication backend

Set up reinforcement learning, which trains models using rewards, with verl. Follow workflows for preparing data, configuring GRPO or PPO training, scaling across GPUs, and addressing memory errors or unstable training.

• Training a math reasoning model with GRPO
• Configuring PPO training with a critic model
• Scaling large model training across multiple nodes

Adapt machine learning training to run across multiple GPUs or machines with Ray Train. Covers changes to existing code, comparing training settings with Ray Tune, and saving checkpoints so training can resume after failures.

• Adapting existing PyTorch code for multiple GPUs
• Training HuggingFace models across multiple machines
• Comparing learning rates and batch sizes with Ray Tune

Adapt PyTorch training scripts for multiple GPUs or machines with Accelerate. This skill guides code changes, configuration, and launch commands, including mixed precision, which combines numerical formats, and DeepSpeed or FSDP integration for distributed training.

• Moving single-GPU training to multiple GPUs
• Configuring mixed precision training
• Setting up DeepSpeed or FSDP

Add PyTorch FSDP2 to training scripts so model state can be split across GPUs. Covers distributed initialization, mixed precision, CPU offloading, and checkpoint saving and loading for models that exceed a single GPU's memory.

• Training a model that exceeds one GPU's memory
• Adding FSDP2 to an existing training script
• Configuring mixed precision and CPU offloading

Organize PyTorch training code with LightningModule and Trainer. This skill guides training, validation, logging, model checkpoint saving, early stopping when progress stalls, and configuration for running the same model across multiple GPUs.

• Organize existing PyTorch training code
• Add validation and metric logging
• Save the best models and stop stalled training

Helps adapt machine learning training to multiple GPUs and machines with Ray Train. Covers resource allocation, recovery from saved training states, and Ray Tune searches that compare settings such as learning rate and batch size.

• Adapt existing PyTorch training to multiple GPUs
• Run HuggingFace training across multiple machines
• Compare learning rates and batch sizes with Ray Tune

Configure large language model training across multiple NVIDIA GPUs with Megatron-Core. The skill helps choose how to split models and workloads, set training parameters, monitor performance, and address memory shortages or low GPU utilization.

• Planning distributed training for a LLaMA-style model
• Distributing MoE expert networks across GPUs
• Tuning GPU utilization and training throughput

Configure large language model training with NVIDIA Megatron-Core across multiple GPUs. This skill helps choose how to split computation, set training parameters, monitor GPU efficiency, and address memory limits or unstable training.

• Planning multi-GPU training for a LLaMA-style model
• Distributing MoE model experts across GPUs
• Tuning training throughput and GPU utilization

Implement and troubleshoot large-model training with DeepSpeed across multiple GPUs. The skill covers memory optimization, parallel training, and numerical precision choices, alongside examples for storage transfers and models that divide work among specialized components.

• Review settings for distributed model training
• Tune transfers between NVMe storage and tensors
• Configure training with specialized model components

Helps configure and troubleshoot DeepSpeed for training large models across multiple devices. Covers memory optimization, parallel training, and DeepNVMe storage transfers, with examples for tuning data movement and configuring models that divide work among specialized components.

• Configuring distributed training for large models
• Tuning tensor transfers to NVMe storage
• Setting up models with specialized expert components

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

The English summary is being prepared. Showing the original description.

Implement and debug large-model training with PyTorch FSDP, which splits training state across devices. Consult documentation and code patterns for sharding choices, mixed precision, and checkpoints when building or reviewing distributed training code.

• Implement distributed training for large models
• Compare FSDP wrapping and sharding patterns
• Review mixed-precision and checkpoint examples

Helps organize PyTorch model training with LightningModule and Trainer, including validation, logging, model saving, and early stopping. It also guides distributed training across multiple GPUs and settings for reduced-precision computation.

• Converting existing PyTorch training code to Lightning
• Adding validation metrics and model checkpoints
• Setting up training across multiple GPUs

Organize PyTorch models and data pipelines with PyTorch Lightning, automate training, save checkpoints, and track experiments. Includes templates and guidance for configuring training across multiple GPUs or TPUs.

• Restructure PyTorch training code into LightningModules
• Build reusable data preparation and loading pipelines
• Configure checkpoints, early stopping, and experiment logs

Guides setup and operation of Lambda Labs GPU instances for machine learning training and inference. Covers SSH access, persistent storage, API automation, and multi-GPU or multi-node training, including how to retain checkpoints and terminate idle instances.

• Setting up a GPU instance for LLM fine-tuning
• Keeping datasets and checkpoints between sessions
• Automating instance launch and termination through the API

Guides setup and operation of Lambda Labs GPU instances for machine learning training and inference. Covers SSH access, persistent storage, API automation, and multi-GPU or multi-node training, including how to retain checkpoints and terminate idle instances.

• Setting up a GPU instance for LLM fine-tuning
• Keeping datasets and checkpoints between sessions
• Automating instance launch and termination through the API