Developers and students with a small GPU who are comfortable with Python and Hugging Face transformers, and who have plenty of free disk space.

What can you build with airllm?

Try a 70B model such as Llama 3 on a laptop or desktop GPU with 4GB of memory. Compare answers from several very large open models without renting big GPUs. Run a model from a Hugging Face ID or a local folder with the familiar generate call. Fine-tune a large model with adapters on a single consumer graphics card. Experiment with 4-bit or 8-bit compression to see how it changes loading speed.

lyogavin /

airllm

Run very large open language models on a small GPU by loading one layer at a time

40.4k stars
5.1k stars since 3 Oct
Jupyter Notebook
Updated yesterday
Some experience
Models & local AI

Open on GitHub (opens in a new tab)

Big language models normally need a GPU large enough to hold all their weights. AirLLM avoids that. It splits a downloaded model into layers saved on disk, then loads one layer onto the GPU, runs it, frees it, and moves to the next. GPU memory depends on the size of a single layer, not the whole model.

You use it much like the Hugging Face transformers library. You pass a model ID to AutoModel.from_pretrained, tokenize your text, and call generate. The README says this works without quantization, distillation or pruning. An optional 4-bit or 8-bit compression only shrinks the weights read from disk, which the README says can speed things up by up to 3x.

The trade-off is disk and time. The split copy of the model needs a lot of disk space, and reading layers from disk at every step is slower than keeping the model in memory. The README gives no speed figures for the largest models, so it looks better suited to experiments than to serving many users.

What it does

One layer on the GPU at a time
Only the layer being computed sits in GPU memory. The README lists Llama 3.x 70B at about 4GB, Llama 3.1 405B at about 8GB and DeepSeek-V3 (671B) at about 12GB.
One AutoModel for many families
Pass a Hugging Face ID or a local path. The README lists Llama, Qwen, DeepSeek, Mistral and Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi and Kimi K3.
Optional 4-bit or 8-bit compression
Set compression to '4bit' or '8bit' (needs bitsandbytes). It quantizes only the weights loaded from disk, which the README says gives up to 3x faster inference with small accuracy loss.
Training with small VRAM
A newer training mode streams frozen weights one layer at a time and keeps adapters on the GPU. The README says a 125B Qwen model trains in under 6GB.
Runs on Mac and CPU
Apple silicon is supported through mlx and torch, and the update notes mention CPU inference. Options include delete_original to save disk space and prefetching to overlap loading with compute.

What you could build

Try a 70B model such as Llama 3 on a laptop or desktop GPU with 4GB of memory.
Compare answers from several very large open models without renting big GPUs.
Run a model from a Hugging Face ID or a local folder with the familiar generate call.
Fine-tune a large model with adapters on a single consumer graphics card.
Experiment with 4-bit or 8-bit compression to see how it changes loading speed.

Get started

Install the package, then call AutoModel.from_pretrained with a Hugging Face model ID and use generate as in the README example. The first run splits the model into layers, so make sure the Hugging Face cache folder has enough disk space.

pip install airllm

Commands are copied from the project's README. Read it on GitHub for the full setup.