jingyaogong /
minimind
Train a tiny 64M-parameter language model from scratch on one consumer GPU
- 64.3k stars
- 1.2k stars since 3 Oct
- Python
- Updated 30 days ago
- Some experience
- Models & local AI
Most people who work with large language models only use them or fine-tune an existing one. Libraries like transformers, trl and peft hide the inner workings behind a few lines of code. MiniMind goes the other way. It implements the core model and training algorithms directly in plain PyTorch, so you can read how each stage works.
The model is small: about 64M parameters, with a 6,400-token vocabulary. The author says one epoch of the SFT stage takes about 2 hours on a single NVIDIA 3090, at a GPU rental cost of about 3 yuan. You download a small dataset, run a pretraining script, then a fine-tuning script, and end up with a chatbot you trained yourself.
The repo goes well beyond basic training. It includes LoRA, DPO, PPO, GRPO and CISPO, tool calling, agentic reinforcement learning, distillation and a mixture-of-experts variant. The architecture follows Qwen3, so the weights work with transformers, llama.cpp, vllm and ollama. The README is mainly in Chinese, with an English version linked.
What it does
Full training pipeline
Scripts cover tokenizer training, pretraining, full SFT, LoRA, DPO, PPO/GRPO/CISPO reinforcement learning, tool use, agentic RL and distillation.
Core code written from scratch
Key algorithms are implemented in native PyTorch rather than through third-party wrappers. The README says LoRA and DPO, for example, do not use peft or trl.
Dense and MoE models
It provides a dense 64M model and a mixture-of-experts model of about 198M parameters (64M active), with structure aligned to Qwen3 and Qwen3-MoE.
Works with common tools
Models are compatible with transformers, llama.cpp, vllm and ollama. It also includes an OpenAI-compatible server and a Streamlit chat page.
Resumable multi-GPU training
Supports single GPU or multi-GPU runs (DDP, DeepSpeed), checkpoint resume, and logging with wandb or SwanLab.
What you could build
- Train your own small chat model end to end on a single gaming-class GPU.
- Study a compact Transformer and MoE implementation line by line to learn how LLMs are built.
- Practice SFT, LoRA, DPO and GRPO on a model small enough to iterate on quickly.
- Fine-tune a tiny model on a private dataset, such as a domain assistant.
- Serve a model you trained through an OpenAI-compatible API for a chat UI such as Open-WebUI.
Get started
Clone the repo and install the requirements. To just try it, download the minimind-3 model and run the CLI chat. To train, download the mini pretrain and SFT datasets into ./dataset, then run the pretraining script followed by the SFT script from the trainer folder.
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simplecd trainer && python train_pretrain.pycd trainer && python train_full_sft.pyCommands are copied from the project's README. Read it on GitHub for the full setup.
More in Models & local AI
transformers
Shared model definitions for text, vision, audio and multimodal AI models
open-webui
Self-hosted chat interface for Ollama and OpenAI-compatible models, with RAG and plugins
odysseus
Self-hosted workspace for AI chat, agents, research, documents, email and calendar
Kronos
Pre-trained Transformer models that forecast candlestick (K-line) price data
airllm
Run very large open language models on a small GPU by loading one layer at a time
Vane
Self-hosted AI answer engine that searches the web and cites its sources
timesfm
Pretrained Google model that forecasts time series without training on your data
Unlimited-OCR
Open OCR model that turns images and multi-page PDFs into structured text in one pass
gallery
Android and iOS app for running open LLMs offline on your phone
lingbot-map
Feed-forward model that builds 3D scenes from video frames as they stream in