# minimind

> Train a tiny 64M-parameter language model from scratch on one consumer GPU

- Repository: https://github.com/jingyaogong/minimind
- Category: Models & local AI
- Stars: 64,255 · Forks: 8,362 · Language: Python · License: Apache-2.0
- Experience: Some experience
- Page: https://ir.adyog.com/repos/jingyaogong/minimind/

Most people who work with large language models only use them or fine-tune an existing one. Libraries like transformers, trl and peft hide the inner workings behind a few lines of code. MiniMind goes the other way. It implements the core model and training algorithms directly in plain PyTorch, so you can read how each stage works.

The model is small: about 64M parameters, with a 6,400-token vocabulary. The author says one epoch of the SFT stage takes about 2 hours on a single NVIDIA 3090, at a GPU rental cost of about 3 yuan. You download a small dataset, run a pretraining script, then a fine-tuning script, and end up with a chatbot you trained yourself.

The repo goes well beyond basic training. It includes LoRA, DPO, PPO, GRPO and CISPO, tool calling, agentic reinforcement learning, distillation and a mixture-of-experts variant. The architecture follows Qwen3, so the weights work with transformers, llama.cpp, vllm and ollama. The README is mainly in Chinese, with an English version linked.

## What it does

- **Full training pipeline**: Scripts cover tokenizer training, pretraining, full SFT, LoRA, DPO, PPO/GRPO/CISPO reinforcement learning, tool use, agentic RL and distillation.
- **Core code written from scratch**: Key algorithms are implemented in native PyTorch rather than through third-party wrappers. The README says LoRA and DPO, for example, do not use peft or trl.
- **Dense and MoE models**: It provides a dense 64M model and a mixture-of-experts model of about 198M parameters (64M active), with structure aligned to Qwen3 and Qwen3-MoE.
- **Works with common tools**: Models are compatible with transformers, llama.cpp, vllm and ollama. It also includes an OpenAI-compatible server and a Streamlit chat page.
- **Resumable multi-GPU training**: Supports single GPU or multi-GPU runs (DDP, DeepSpeed), checkpoint resume, and logging with wandb or SwanLab.

## What you could build

- Train your own small chat model end to end on a single gaming-class GPU.
- Study a compact Transformer and MoE implementation line by line to learn how LLMs are built.
- Practice SFT, LoRA, DPO and GRPO on a model small enough to iterate on quickly.
- Fine-tune a tiny model on a private dataset, such as a domain assistant.
- Serve a model you trained through an OpenAI-compatible API for a chat UI such as Open-WebUI.

## Who it's for

Students and developers who want to understand how LLMs are trained, who know basic Python and PyTorch and have access to an NVIDIA GPU or a rented one.

## Get started

Clone the repo and install the requirements. To just try it, download the minimind-3 model and run the CLI chat. To train, download the mini pretrain and SFT datasets into ./dataset, then run the pretraining script followed by the SFT script from the trainer folder.

```
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
```
```
cd trainer && python train_pretrain.py
```
```
cd trainer && python train_full_sft.py
```

Tags: llm, pytorch, training-from-scratch, tutorial, moe