rasbt /
LLMs-from-scratch
Code for the book that walks you through building a GPT-style model in PyTorch
- 107.1k stars
- Jupyter Notebook
- Updated 20 days ago
- Some experience
- Learning & resource lists
Most people use LLMs as black boxes. This repository is the code companion to the book Build a Large Language Model (From Scratch). It lets you open the box by coding each stage yourself, chapter by chapter, in Jupyter notebooks.
You start with text data and tokenizing, then write attention mechanisms, then assemble a GPT model. Next you pretrain it on unlabeled text and finetune it for classification and for following instructions. The model is small and meant for learning, but the README says the method mirrors how large foundation models are made. The code also shows how to load weights from larger pretrained models for finetuning.
It uses plain PyTorch and no external LLM libraries, so nothing hides behind an API. The main chapters are designed to run on an ordinary laptop and use a GPU automatically if one is present. Bonus folders go further, with topics such as KV cache, grouped-query attention, mixture-of-experts and LoRA. A video course and a free quiz PDF are linked as companions.
What it does
Chapter-by-chapter notebooks
Each chapter has a main notebook, a condensed script or summary notebook, and exercise solutions. Topics run from text data and attention to GPT, pretraining, classification finetuning and instruction finetuning.
No external LLM libraries
Everything is built in PyTorch from the ground up, so you can read every step of the model and training loop.
Runs on a laptop
The main chapter code is designed for conventional hardware and uses a GPU automatically when available.
Bonus material
Optional folders cover a BPE tokenizer from scratch, KV cache, grouped-query attention, mixture-of-experts, LoRA and more.
Practice and companions
Exercise solutions are included, with a free quiz PDF, a 17-hour video course and a sequel book on reasoning models.
What you could build
- Learn how attention and transformer blocks work by coding them yourself.
- Pretrain a small GPT-style model on a text dataset such as Project Gutenberg.
- Finetune a pretrained GPT model to classify text or follow instructions.
- Add LoRA parameter-efficient finetuning to a training loop.
- Follow along as a self-study course or a university class on LLMs.
Get started
Clone the repository, then set up Python and the packages by following the setup folder's README. Open the chapter 2 notebook and work through the chapters in order.
git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.gitCommands are copied from the project's README. Read it on GitHub for the full setup.
More in Learning & resource lists
ai-engineering-from-scratch
Free 20-phase course that teaches AI engineering by building each piece by hand
AI-For-Beginners
A free 12-week, 24-lesson course on AI with notebooks, quizzes and labs
system_prompts_leaks
A library of system prompts from popular AI chatbots and coding tools
dive-into-llms
Free Chinese notebooks and slides for learning to fine-tune, prompt and secure LLMs
awesome-claude-code
A curated list of guides, skills, tools and plugins for Claude Code users
OpenMAIC
Turns a topic or document into an interactive classroom taught by AI agents
12-factor-agents
A written guide to twelve principles for building reliable LLM-powered software
easy-vibe
A free, multilingual course that teaches AI-assisted coding by building real products
awesome-claude-skills
A curated list of Claude Skills, grouped by task, with links to each one
reasoning-from-scratch
Code for a book that adds reasoning to a small Qwen3 model in PyTorch