What can you build with autoresearch?

Run an overnight loop of automated experiments on a small GPT and read the log in the morning. Write and refine program.md to test which agent instructions produce faster research progress. Study a compact GPT training script with the Muon and AdamW optimizers. Try the setup on a smaller machine by shrinking the model and using a narrow dataset like TinyStories. Fork it to support another platform or add more agents to the research setup.

karpathy /

autoresearch

Let an AI agent edit a small LLM training script overnight and keep what improves it

98.4k stars
1.3k stars since 3 Oct
Python
Updated 7 months ago
Some experience
AI agents

Open on GitHub (opens in a new tab)

Machine learning research usually means a person tweaks code, trains a model, reads the result, and tries again. This project hands that loop to an AI agent. It uses a simplified single-GPU version of nanochat, a small LLM training codebase, as the thing being improved.

The repo has three files that matter. prepare.py downloads data, trains a tokenizer, and holds the evaluation code, and it stays fixed. train.py holds the GPT model, the optimizer (Muon and AdamW), and the training loop, and the agent is free to rewrite it. program.md holds the instructions for the agent, and that is the file you edit.

Every run gets the same 5-minute wall-clock budget and is scored by val_bpb (validation bits per byte), where lower is better. Because the budget and metric are fixed, results are comparable whatever the agent changes. The catch is that results are not comparable across different hardware.

What it does

Agent edits one file
The agent only changes train.py: architecture, hyperparameters, optimizer, batch size. Keeping it to one file keeps diffs easy to review.
Fixed 5-minute runs
Each experiment trains for exactly five minutes, excluding startup. That works out to about 12 runs an hour, or around 100 overnight.
Single comparable metric
val_bpb does not depend on vocabulary size, so architecture changes are judged fairly against each other.
Instructions live in program.md
You steer the research by editing a Markdown file, which the README calls a lightweight skill. The default is a bare-bones baseline for one agent.
Tips for smaller machines
The README lists settings to lower for weaker hardware, such as DEPTH, MAX_SEQ_LEN and TOTAL_BATCH_SIZE. It also links forks for macOS, Windows and AMD.

What you could build

Run an overnight loop of automated experiments on a small GPT and read the log in the morning.
Write and refine program.md to test which agent instructions produce faster research progress.
Study a compact GPT training script with the Muon and AdamW optimizers.
Try the setup on a smaller machine by shrinking the model and using a narrow dataset like TinyStories.
Fork it to support another platform or add more agents to the research setup.

Get started

You need a single NVIDIA GPU (tested on H100), Python 3.10+ and uv. Install dependencies, run prepare.py once to download data and train the tokenizer, then run uv run train.py to check one 5-minute experiment works. After that, open a coding agent in the repo and ask it to read program.md and start an experiment.

curl -LsSf https://astral.sh/uv/install.sh | sh

uv sync

uv run prepare.py

Commands are copied from the project's README. Read it on GitHub for the full setup.

Who it's for

Suits ML-curious developers and researchers with an NVIDIA GPU who are comfortable running a coding agent; knowing basic neural network training helps.

Useful for

ML engineers and data scientistsCompact GPT training script with Muon and AdamW, plus an overnight agent loop that cuts validation loss.
AI engineersShows autonomous agent experimentation steered by a single program.md file.
Students and beginnersSmall readable training codebase for learning how LLM training and automated research work.