Building a GPT-style transformer from scratch

Most from-scratch tutorials stop at the forward pass. This repo deliberately covers the full pipeline so you see how a model is actually produced, not just defined. It starts with downloading The Pile and preprocessing a subset, then tokenizes with tiktoken's r50k_base encoding (the GPT-3/ChatGPT vocabulary), inserts <|endoftext|> at document boundaries, and writes the token stream to HDF5 for fast streaming during training ^[1]. The model itself is built up from primitives: single and multi-head causal self-attention (16 heads by default, concatenated through a projection layer), a 4x-expansion ReLU MLP feed-forward block, and stacked transformer blocks wired with layer norm and residual connections ^[1]. On top of that sits a training loop with periodic evaluation, learning-rate decay, and crash-safe checkpoints. The payoff of teaching the entire chain is that every black box gets opened: a learner who finishes the repo can point to the exact line where attention scores are masked, where embeddings are looked up, and where logits become sampled text.

The headline shift is economic. A ~13M-parameter model trains within one day on a single free Tesla T4 (16GB) on Colab or Kaggle, and that is roughly the scale where output starts producing correct grammar and spelling ^[1]. The repo's own scale table shows how far the same code stretches with more hardware: a T4 handles ~13M+ parameters, an RTX 4090 (24GB) reaches ~4B, and an A100 (40GB) reaches ~6B-8B ^[1]. Karpathy's build-nanogpt makes the same point from the other direction: reproducing GPT-2 (124M) is now, in his words, 'a matter of ~1hr and ~$10' ^[2]. But the ceiling is real and steep. The repo's training corpus is a 5-10% slice of an 825GB dataset ^[1][3], and frontier-scale models need orders of magnitude more data and compute than a free notebook provides. What collapsed is the cost of building a working transformer end to end; what did not collapse is the cost of building a competitive one.

The implicit argument of the project is pedagogical: writing the attention, multi-head projection, feed-forward, embedding, residual, and layer-norm code yourself is the only way to see how they fit together, rather than calling a high-level wrapper that hides them ^[1]. This mirrors the design of Karpathy's nanoGPT and build-nanogpt, which teach GPT as clean, readable PyTorch precisely so the architecture is legible rather than abstracted away ^[2][4]. Community reception in learning circles is positive but clear-eyed about the trade-off: the value is education, not deployment. The honest framing from practitioners is that you will learn more about backpropagation and the mechanics of training than about LLMs as products, and that building it yourself instead of loading a pretrained checkpoint is worth it specifically for the understanding it buys.

Not everyone agrees the exercise is the right move for a working product. The sharpest community critique is that a tiny from-scratch model is not useful on your own data: if you actually want a model that performs on a custom task, supervised fine-tuning of an open base model is the more efficient path than pretraining 13M parameters from zero. Related objections note that vanilla transformers are data-inefficient compared with modern architectures, that the setup can be tied to a narrow Linux/CUDA environment, and that the from-scratch genre arrives years after the canonical teaching material. The repo partly answers this by extending into post-training, shipping guides for SFT, a reward model, PPO, DPO, and GRPO so the journey does not end at a raw pretrained base ^[1]. The resolution is to be honest about intent: build from scratch to understand the machine, fine-tune an open model to get a useful one.