Results for “sdp”

Get guidance on distributed training with PyTorch FSDP, which splits model parameters across devices. The skill covers implementation, mixed precision, moving data to CPU memory, communication setup, and debugging distributed training code.

• Implement distributed training with FSDP
• Review mixed precision and CPU offloading settings
• Choose a distributed communication backend

Add PyTorch FSDP2 to training scripts so model state can be split across GPUs. Covers distributed initialization, mixed precision, CPU offloading, and checkpoint saving and loading for models that exceed a single GPU's memory.

• Training a model that exceeds one GPU's memory
• Adding FSDP2 to an existing training script
• Configuring mixed precision and CPU offloading

Implement and debug large-model training with PyTorch FSDP, which splits training state across devices. Consult documentation and code patterns for sharding choices, mixed precision, and checkpoints when building or reviewing distributed training code.

• Implement distributed training for large models
• Compare FSDP wrapping and sharding patterns
• Review mixed-precision and checkpoint examples

Adapt PyTorch training scripts for multiple GPUs or machines with Accelerate. This skill guides code changes, configuration, and launch commands, including mixed precision, which combines numerical formats, and DeepSpeed or FSDP integration for distributed training.

• Moving single-GPU training to multiple GPUs
• Configuring mixed precision training
• Setting up DeepSpeed or FSDP

Deploy a Software-Defined Perimeter that connects verified users to authorized applications. Configure single-packet authorization, mutual certificate authentication, access policies, and monitoring, then test whether unauthorized connections and access between applications are blocked.

• Building zero trust access to internal applications
• Hiding services from unauthenticated network scans
• Testing certificate authentication and access restrictions

Adapt existing PyTorch training scripts to use multiple GPUs or machines with limited code changes. The skill guides configuration, launch commands, mixed precision, which uses lower-precision numbers, and DeepSpeed or FSDP settings for distributing training work and memory.

• Move a single-GPU training script to multiple GPUs
• Configure mixed precision training
• Set up DeepSpeed or FSDP to reduce memory use

Adapt existing PyTorch training scripts to run across multiple GPUs or machines with Accelerate. This skill covers device setup, distributed training configuration, and mixed precision, which uses lower-precision numbers to reduce memory use.

• Adapt a single-GPU training script for multiple GPUs
• Configure mixed precision to reduce training memory use
• Set up training with DeepSpeed or FSDP

Optimize transformer attention, the calculation that relates tokens to one another, using Flash Attention. The skill guides PyTorch integration, GPU memory reduction, performance measurement, and output comparisons for training or inference with long sequences.

• Reducing GPU memory use with long sequences
• Integrating Flash Attention into PyTorch models
• Comparing runtime and outputs before and after changes

Orientação especializada para treinamento com Fully Sharded Data Parallel do PyTorch (FSDP) - sharding de parâmetros, precisão mista, CPU offloading, FSDP2

The English summary is being prepared. Showing the original description.

Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP - parameter sharding, mixed precision, CPU offloading, FSDP2

The English summary is being prepared. Showing the original description.

Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP - parameter sharding, mixed precision, CPU offloading, FSDP2

The English summary is being prepared. Showing the original description.

Integrate Flash Attention into transformer models to reduce processing time and GPU memory use for long sequences. Covers PyTorch integration, advanced attention options, and benchmarks that compare speed, memory use, and output accuracy.

• Speeding up transformer training on long sequences
• Reducing GPU memory use during attention
• Adding Flash Attention to a PyTorch model

Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.

• Pretraining Llama 3.1 8B on eight GPUs
• Configuring multi-node training with SLURM
• Enabling Float8 training on H100 GPUs

Configure large language model pretraining with torchtitan across multiple GPUs. Covers ways to split training work, launch jobs on computing clusters, enable Float8 numerical processing, and save or resume training through distributed checkpoints.

• Pretraining Llama 3.1 8B on eight GPUs
• Configuring multi-node training with SLURM
• Enabling Float8 training on H100 GPUs

Organize PyTorch training code with LightningModule and Trainer. This skill guides training, validation, logging, model checkpoint saving, early stopping when progress stalls, and configuration for running the same model across multiple GPUs.

• Organize existing PyTorch training code
• Add validation and metric logging
• Save the best models and stop stalled training

Assess Bluetooth Classic devices such as older car kits, industrial sensors, and audio accessories. The skill guides device discovery, service inspection, authentication checks, and known vulnerability assessment, helping document exposed functions and device-specific findings.

• Assessing older in-car Bluetooth devices
• Checking authentication on industrial device services
• Reviewing exposure to BlueBorne and KNOB

Official NVIDIA-authored guidance for PhysicsNeMo ShardTensor domain parallelism — integrate domain parallelism into training/inference scripts (new or existing) with DDP or FSDP2, write and register shard patches to enable new layers/ops, and bootstrap multi-GPU correctness tests. Use when working with ShardTensor, scatter_tensor, domain parallelism, sequence/spatial sharding, ring attention, DeviceMesh + DDP/FSDP2 hybrid parallelism, or physicsnemo.domain_parallel. Do NOT use for generic PyTorch DDP/FSDP setup without domain parallelism, picking a PhysicsNeMo model or example (use physicsnemo-discover), or non-distributed training questions.

The English summary is being prepared. Showing the original description.

Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP - parameter sharding, mixed precision, CPU offloading, FSDP2

The English summary is being prepared. Showing the original description.

Organize PyTorch models and data pipelines with PyTorch Lightning, automate training, save checkpoints, and track experiments. Includes templates and guidance for configuring training across multiple GPUs or TPUs.

• Restructure PyTorch training code into LightningModules
• Build reusable data preparation and loading pipelines
• Configure checkpoints, early stopping, and experiment logs

Organize PyTorch training code with LightningModule and Trainer. Follow examples for validation, logging, model saving, and distributed training across multiple GPUs, with callbacks that add monitoring or stop training when progress stalls.

• Converting an existing PyTorch training loop
• Adding validation, logs, and model checkpoints
• Configuring training across multiple GPUs

Operational guide for enabling Megatron FSDP in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.

The English summary is being prepared. Showing the original description.

Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.

The English summary is being prepared. Showing the original description.

Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.

The English summary is being prepared. Showing the original description.

Install Triton + SageAttention to accelerate ComfyUI (the sageattn attention_mode and inductor torch.compile used by WanVideoWrapper / many video graphs). Windows-first (triton-windows + woct0rdho prebuilt SageAttention wheels matched to torch/CUDA/python into the RIGHT python), plus Linux (official triton + build) and Mac (N/A → sdpa/MPS). Also covers the SAFE sdpa / no-compile fallback so an example that assumes sageattn + torch.compile still runs when these aren't installed (video-extend TRAP 5). Use when a loader crashes with "No module named 'sageattention'" or reports triton unavailable, when asked to speed up Wan/video workflows, or when deciding whether to install acceleration vs. fall back.

The English summary is being prepared. Showing the original description.