leanctx

Long prompts cost money. RAG apps, chat agents and coding agents keep sending large documents, logs and old turns to the model. leanctx sits between your code and the provider and trims that input. You change one import line and the rest of your code stays the same.

It classifies each segment of a request. Code, stack traces, JSON and tool-call fields pass through untouched. Prose such as documentation, retrieved passages, logs and earlier turns goes through LLMLingua-2, a small model that runs on your machine and drops less useful words. An optional mode asks a cheap LLM to summarize instead. After rebuilding the request it checks invariants, and if anything fails or times out it sends the original.

The README compares it with three alternatives. Provider prompt caching helps with stable prefixes but not changing content, so it suggests using both. Naive truncation cuts the middle of documents. Hosted compression APIs need your data sent to them, while leanctx runs locally. Savings depend on how much of your traffic is prose. On LongBench v2, layered on top of another compressor, it reports 18.7% fewer tokens for a 1.8 point accuracy drop. It reports about 37% savings on prose-heavy traffic and about 10% on code-heavy traffic.