# LLMs-from-scratch

> Code for the book that walks you through building a GPT-style model in PyTorch

- Repository: https://github.com/rasbt/LLMs-from-scratch
- Category: Learning & resource lists
- Stars: 107,129 · Forks: 16,491 · Language: Jupyter Notebook · License: not stated
- Experience: Some experience
- Page: https://ir.adyog.com/repos/rasbt/llms-from-scratch/

Most people use LLMs as black boxes. This repository is the code companion to the book Build a Large Language Model (From Scratch). It lets you open the box by coding each stage yourself, chapter by chapter, in Jupyter notebooks.

You start with text data and tokenizing, then write attention mechanisms, then assemble a GPT model. Next you pretrain it on unlabeled text and finetune it for classification and for following instructions. The model is small and meant for learning, but the README says the method mirrors how large foundation models are made. The code also shows how to load weights from larger pretrained models for finetuning.

It uses plain PyTorch and no external LLM libraries, so nothing hides behind an API. The main chapters are designed to run on an ordinary laptop and use a GPU automatically if one is present. Bonus folders go further, with topics such as KV cache, grouped-query attention, mixture-of-experts and LoRA. A video course and a free quiz PDF are linked as companions.

## What it does

- **Chapter-by-chapter notebooks**: Each chapter has a main notebook, a condensed script or summary notebook, and exercise solutions. Topics run from text data and attention to GPT, pretraining, classification finetuning and instruction finetuning.
- **No external LLM libraries**: Everything is built in PyTorch from the ground up, so you can read every step of the model and training loop.
- **Runs on a laptop**: The main chapter code is designed for conventional hardware and uses a GPU automatically when available.
- **Bonus material**: Optional folders cover a BPE tokenizer from scratch, KV cache, grouped-query attention, mixture-of-experts, LoRA and more.
- **Practice and companions**: Exercise solutions are included, with a free quiz PDF, a 17-hour video course and a sequel book on reasoning models.

## What you could build

- Learn how attention and transformer blocks work by coding them yourself.
- Pretrain a small GPT-style model on a text dataset such as Project Gutenberg.
- Finetune a pretrained GPT model to classify text or follow instructions.
- Add LoRA parameter-efficient finetuning to a training loop.
- Follow along as a self-study course or a university class on LLMs.

## Who it's for

Suits learners and engineers who want to understand LLM internals; you should be comfortable with Python, and PyTorch basics help (Appendix A covers them).

## Get started

Clone the repository, then set up Python and the packages by following the setup folder's README. Open the chapter 2 notebook and work through the chapters in order.

```
git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git
```

Tags: llm, pytorch, gpt, tutorial, transformers