Episode 68

How GPT-2 Is Trained

Next-token prediction on huge corpora with a decoder-only stack.

• Causal LM objective
• BPE tokenization
• Decoder-only architecture
• LR warmup & schedules

Notes for this episode are coming soon.