13 skills · sorted by relevance
Learn how GPT language models work by preparing text, training a small model, and generating samples. Covers CPU experiments, custom datasets, further training of pretrained GPT-2 models, and reproducing GPT-2 with multiple GPUs.
Learn how GPT language models work by preparing text, training a small model, and generating samples. Covers CPU experiments, custom datasets, further training of pretrained GPT-2 models, and reproducing GPT-2 with multiple GPUs.
Investigate how transformer language models produce predictions with TransformerLens. This skill guides inspection of intermediate values and attention patterns, plus experiments that replace internal values to test their effects and identify components involved in model behavior.
Investigate how transformer language models produce predictions with TransformerLens. This skill guides inspection of intermediate values and attention patterns, plus experiments that replace internal values to test their effects and identify components involved in model behavior.
Implementação educacional do GPT em ~300 linhas. Reproduz o GPT-2 (124M) em OpenWebText. Código limpo e hackeável para aprender transformers. Por Andrej Karpathy. Perfeito para entender a arquitetura do GPT do zero. Treine em Shakespeare (CPU) ou OpenWebText (multi-GPU).
The English summary is being prepared. Showing the original description.
Explore concepts learned by language models using sparse autoencoders, which separate internal activity into a small set of features. Analyze pretrained models, train custom autoencoders, assess reconstruction quality, and experiment with feature based output adjustments.
Work with Waypoint models to analyze microbiome samples, the communities of microbes in a sample. Convert taxonomic abundance tables, generate numerical sample representations, configure further training for prediction tasks, and evaluate models on the Compass benchmark.
Guides experiments that inspect and change internal model values using nnsight. Covers local PyTorch models and remote experiments through NDIF, including replacing intermediate values to test how particular layers affect a model’s predictions.
Use SAELens to train and analyze sparse autoencoders, which separate neural network activity into features that may represent recognizable concepts. Explore pretrained features, evaluate reconstruction quality, and investigate how individual features influence language model outputs.
Investigate why a PyTorch model behaves a certain way by changing its internal values with pyvene. The skill guides experiments that swap or remove activations, train interventions, compare outputs, and save or share intervention configurations.
Guides experiments that inspect and modify internal values in PyTorch models to study their effects on predictions. Covers local analysis and remote experiments through NDIF for models too large for local GPUs.
Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.
The English summary is being prepared. Showing the original description.
Guides multi-GPU pre-training: DDP, FSDP2, ZeRO, tensor/pipeline/expert parallelism, fp8/Muon. Use when scaling a run, training MoE, or reproducing GPT-2 on rented GPUs.
The English summary is being prepared. Showing the original description.