# HyperQwen

> vLLM patches and Docker setups to run Qwen3.8-27B fast on one 24 GB GPU

- Repository: https://github.com/syv-ai/HyperQwen
- Category: Models & local AI
- Stars: 2,192 · Forks: 295 · Language: Python · License: Apache-2.0
- Experience: Some experience
- Page: https://ir.adyog.com/repos/syv-ai/hyperqwen/

Large Qwen models are hard to serve well on a card you can buy in a shop. Stock vLLM leaves speed and context length unused on a 24 GB GPU. HyperQwen packages the fixes as Docker Compose profiles, so you can start a server without assembling the pieces yourself.

It has three parts. A series of patches against a pinned vLLM 0.31.0. A pipeline that requantizes the model, including its output heads, and builds a 40k-token draft vocabulary calibrated on the model's own output. And speculative decoding that can draft tokens straight from the prompt, which is why answers that quote the input run much faster.

There are two modes, and one GPU runs one at a time. Single mode suits one or a few people chatting. Batch mode suits API backends with many concurrent requests. The README reports 127 tok/s single-stream, about 1,035 tok/s aggregate at 64 concurrent, and 381 tok/s when the answer quotes the prompt. These numbers come from one RTX 3090 capped at 250 W. Other cards are mostly community reports, and the README lists open faults on some.

## What it does

- **Two ready-made serving modes**: Compose profiles `single` and `batch` start an OpenAI-compatible API with key auth. The port binds to 127.0.0.1 until you set a key and change BIND.
- **Five setups chosen in .env**: A flowchart and table map your use to a setup: batch, default chat, prompt-quoting, long context to 150k tokens, or huge context around 240k.
- **Speculative decoding from the prompt**: Single mode proposes 7 drafts per pass and verifies 15 tokens per step off the context. That gives 381 tok/s when answers quote the prompt.
- **Patch series and requant pipeline**: Patches against vLLM 0.31.0 are listed one line each in PATCHES.md. The prepare step requantizes the model into ./models on first start (about 20 GB).
- **Field-test protocol**: A benchmark script and a field-report form let owners of other GPUs add comparable results. The README names which cards are still missing.

## What you could build

- Run a private OpenAI-compatible Qwen3.8-27B endpoint on a home RTX 3090.
- Serve a batch pipeline that sends many concurrent requests to one local GPU.
- Speed up code edits, RAG answers, rewrites or translation, where output repeats the input.
- Work with prompts up to 150k tokens on a 24 GB card.
- Benchmark your own GPU and file a comparable field report.

## Who it's for

Suits people with an NVIDIA GPU of 24 GB or so who are comfortable with Docker and know what vLLM and an OpenAI-style API are.

## Get started

Clone the repo, copy .env.example to .env, then start one profile with Docker Compose. The first start pulls a 9.5 GB image and requantizes the model, then serves on port 18020. Set VLLM_API_KEY and BIND before exposing it beyond your machine.

```
git clone https://github.com/syv-ai/HyperQwen && cd HyperQwen
```
```
cp .env.example .env
```
```
docker compose --profile single up -d
```

Tags: vllm, qwen, local-llm, quantization, speculative-decoding