Results for “nanogpt”

Learn how GPT language models work by preparing text, training a small model, and generating samples. Covers CPU experiments, custom datasets, further training of pretrained GPT-2 models, and reproducing GPT-2 with multiple GPUs.

• Train a small model on Shakespeare using a CPU
• Prepare custom text for model training
• Further train a pretrained GPT-2 model

Learn how GPT language models work by preparing text, training a small model, and generating samples. Covers CPU experiments, custom datasets, further training of pretrained GPT-2 models, and reproducing GPT-2 with multiple GPUs.

• Train a small model on Shakespeare using a CPU
• Prepare custom text for model training
• Further train a pretrained GPT-2 model

Implementação educacional do GPT em ~300 linhas. Reproduz o GPT-2 (124M) em OpenWebText. Código limpo e hackeável para aprender transformers. Por Andrej Karpathy. Perfeito para entender a arquitetura do GPT do zero. Treine em Shakespeare (CPU) ou OpenWebText (multi-GPU).

The English summary is being prepared. Showing the original description.

Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.

The English summary is being prepared. Showing the original description.

Builds a GPT and BPE tokenizer from scratch. Use when implementing autograd, attention, a nanoGPT loop, muP hyperparameter transfer, WSD annealing, loss spikes, or mid-training.

The English summary is being prepared. Showing the original description.