syv-ai /
HyperQwen
vLLM patches and Docker setups to run Qwen3.8-27B fast on one 24 GB GPU
- 2.2k stars
- Python
- Updated yesterday
- Some experience
- Models & local AI
Large Qwen models are hard to serve well on a card you can buy in a shop. Stock vLLM leaves speed and context length unused on a 24 GB GPU. HyperQwen packages the fixes as Docker Compose profiles, so you can start a server without assembling the pieces yourself.
It has three parts. A series of patches against a pinned vLLM 0.31.0. A pipeline that requantizes the model, including its output heads, and builds a 40k-token draft vocabulary calibrated on the model's own output. And speculative decoding that can draft tokens straight from the prompt, which is why answers that quote the input run much faster.
There are two modes, and one GPU runs one at a time. Single mode suits one or a few people chatting. Batch mode suits API backends with many concurrent requests. The README reports 127 tok/s single-stream, about 1,035 tok/s aggregate at 64 concurrent, and 381 tok/s when the answer quotes the prompt. These numbers come from one RTX 3090 capped at 250 W. Other cards are mostly community reports, and the README lists open faults on some.
What it does
Two ready-made serving modes
Compose profiles
singleandbatchstart an OpenAI-compatible API with key auth. The port binds to 127.0.0.1 until you set a key and change BIND.Five setups chosen in .env
A flowchart and table map your use to a setup: batch, default chat, prompt-quoting, long context to 150k tokens, or huge context around 240k.
Speculative decoding from the prompt
Single mode proposes 7 drafts per pass and verifies 15 tokens per step off the context. That gives 381 tok/s when answers quote the prompt.
Patch series and requant pipeline
Patches against vLLM 0.31.0 are listed one line each in PATCHES.md. The prepare step requantizes the model into ./models on first start (about 20 GB).
Field-test protocol
A benchmark script and a field-report form let owners of other GPUs add comparable results. The README names which cards are still missing.
What you could build
- Run a private OpenAI-compatible Qwen3.8-27B endpoint on a home RTX 3090.
- Serve a batch pipeline that sends many concurrent requests to one local GPU.
- Speed up code edits, RAG answers, rewrites or translation, where output repeats the input.
- Work with prompts up to 150k tokens on a 24 GB card.
- Benchmark your own GPU and file a comparable field report.
Get started
Clone the repo, copy .env.example to .env, then start one profile with Docker Compose. The first start pulls a 9.5 GB image and requantizes the model, then serves on port 18020. Set VLLM_API_KEY and BIND before exposing it beyond your machine.
git clone https://github.com/syv-ai/HyperQwen && cd HyperQwencp .env.example .envdocker compose --profile single up -dCommands are copied from the project's README. Read it on GitHub for the full setup.
More in Models & local AI
transformers
Shared model definitions for text, vision, audio and multimodal AI models
open-webui
Self-hosted chat interface for Ollama and OpenAI-compatible models, with RAG and plugins
odysseus
Self-hosted workspace for AI chat, agents, research, documents, email and calendar
minimind
Train a tiny 64M-parameter language model from scratch on one consumer GPU
Kronos
Pre-trained Transformer models that forecast candlestick (K-line) price data
airllm
Run very large open language models on a small GPU by loading one layer at a time
Vane
Self-hosted AI answer engine that searches the web and cites its sources
timesfm
Pretrained Google model that forecasts time series without training on your data
Unlimited-OCR
Open OCR model that turns images and multi-page PDFs into structured text in one pass
gallery
Android and iOS app for running open LLMs offline on your phone