leanctx

leanctx

Drop-in prompt compression for production LLM applications. Cut your input-token bill by 10–40%，without sacrificing accuracy.

# before
from openai import OpenAI

# after
from leanctx import OpenAI  # same interface, compressed requests

On the full LongBench v2 set (N=503), layered on top of a compressor that is already running, leanctx removes an extra 18.7 % of tokens — rising to 36.7 % on prose-heavy traffic — at a cost of 1.8 pp of accuracy. Every figure regenerates from per-item records committed to this repo. Open-source models, runs locally, MIT-licensed. Your prompts and user data never leave your infrastructure by default.

Quickstart · What makes it different · Benchmarks · Integrations · How it works

What makes it different

Loss-tolerance routing — classify every segment of a prompt by how much distortion it can survive, then compress each class differently.

Compression applied uniformly to a prompt will eventually damage the one part that cannot survive being touched, and will do so unpredictably. The hard part is not shortening text — it is deciding, inside a live request, what may be shortened at all.

Quickstart (60 seconds)

pip install 'leanctx[openai,lingua]'    # or [anthropic], [gemini]

from leanctx import OpenAI

client = OpenAI(
    leanctx_config={
        "mode": "on",
        "trigger": {"threshold_tokens": 2000},
        "routing": {"prose": "lingua"},  # route prose through LLMLingua-2
    },
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    max_tokens=512,
    messages=[{"role": "user", "content": LONG_DOCUMENT}],
)

print(response.usage.leanctx_tokens_saved)  # e.g. 1841
print(response.usage.leanctx_ratio)         # e.g. 0.49

First Lingua call loads ~1.2 GB of model weights to ~/.cache/huggingface/. Subsequent calls reuse the cache. Add pip install 'leanctx[lingua]' to opt in; without it, leanctx falls back to passthrough.

Verify the install with no API key needed:

leanctx bench list                                   # 7 registered scenarios
leanctx bench run agent-structural --workload agent  # 5 invariants enforced, exit 0 = pass

Why this exists

You’re building a production LLM app and your token bill is a line item:

• RAG apps with large retrieved documents
• Long-running conversational agents (LangChain / LangGraph / CrewAI)
• Document-processing pipelines
• Coding agents — Cursor-like / Claude-Code-like, with growing tool-call histories

Existing options have gaps:

• Provider prompt caching (Anthropic / OpenAI / Gemini) wins on stable prefixes — system prompts, tool definitions, retrieved-document pools. It doesn’t help with dynamic per-query content (chat history, freshly retrieved docs, tool outputs). Compose with leanctx, don’t choose between them.
• Naive truncation drops the middle of the document, exactly where many answers live. The LongBench v2 numbers above show this concretely.
• Hosted compression APIs (Compresr, Token Company) require sending your context to their servers. Closed-source models. leanctx is MIT-licensed, runs the model locally, and never makes outbound calls except to your existing provider.

Real numbers

Full LongBench v2 sweep — N=503, layered on a production compressor

The headline result. leanctx runs as a semantic pass on top of ClawRouter’s seven structural compression layers, so the measured delta is what leanctx adds to a system that is already compressing. All 503 LongBench v2 questions (Tsinghua KEG, 8K–2M words), Claude Haiku 4.5 eval, temperature 0.1.

Compression costs accuracy, it does not add it. The claim is that the cost is small and bounded — 1.8 pp overall, against the −2 pp go/no-go gate set for the integration. Because 54.2 % of Layer-8 input tokens route to verbatim, the compression actually applied to eligible content is 40.8 %; the verbatim half contributes exactly 0 to both the savings and the accuracy delta by construction.

Sub-buckets are not uniform, and the report gives all of them: short/lingua is −17.6 pp (N=68) and Single-Document QA −11.5 pp (N=78), while Long Structured Data Understanding is +9.4 pp (N=32). Sidecar latency 47 ms p50 on GPU. At Sonnet input pricing the savings are ~$78 per 1,000 requests.

Full report — per-bucket breakdowns by route × difficulty × length × domain, layer-by-layer contributions, cost model: benchmarks/clawrouter/full_long_bench_evaluation_result.md.

Independent execution and audit

This result was not produced by the maintainer. The benchmark was executed and audited by outside contributors (@YingjingLu, @QianXiaoMoRan9). An audit of an earlier draft showed that a headline “+7.4 % on long context” was eval noise rather than a compression effect — 7 of the 8 net improved items came from the verbatim subset, where the input was byte-identical (McNemar p = 0.143). Two methodology fixes followed (#7): Leg B now reuses Leg A’s answer whenever the compressed context is byte-identical, so verbatim items contribute Δ = 0 by construction instead of decoder noise, and eval temperature dropped to 0.1. A separate correction forced a letter choice in the closed-book control, which had been scoring 6 % — below the 25 % random floor for 4-way multiple choice — and inflating apparent context lift.

The numbers above are post-correction. Discussion: issue #3.

Every figure here regenerates from committed data

The 503 per-item records are in the repo, so nothing above has to be taken on trust. The sweep was re-run from a clean checkout on separate hardware, at the documented defaults (llmlingua-2-xlm-roberta-large-meetingbank, ratio 0.5, threshold 1500), and the corpus-level accounting lands on the same values:

Savings are hardware- and configuration-sensitive: the CPU re-run measures 18.7 % blended / 40.8 % on eligible content, against 24.1 % / 52.8 % on the original GPU run. Two effects are visible in the records — 7 long items exceed the connector’s 60 s timeout and fail open under CPU inference (a GPU run stays well inside it), and per-item compression declines as input grows, from 50.0 % on the smallest quartile to 37.7 % on the largest. The figures quoted throughout this README are the conservative ones.

Records, per-figure breakdown, and the script that produces them: results/full503_phase1_results.jsonl · short_route_counterfactual.md · benchmarks/clawrouter/short_route_counterfactual.py.

Savings as a function of traffic mix

The 503-item corpus is 45.8 % prose by token share, and blended savings scale linearly with that share — LLMLingua-2 is deterministic and extractive, so reweighting the per-item records to any target mix is exact arithmetic rather than resampling:

blended_savings(p) = p × r        p = prose token-share,  r = 40.8 % (95 % CI 37.8–43.7)

So “what will leanctx save me?” has one honest answer: it depends on how much of your traffic is prose, and the curve above tells you. Full curve and CSV: prose_code_savings_curve.md.

Ablation vs naive truncation — 15 items, directional

A separate, much smaller comparison against head+tail truncation at a 20K cap. Same model, same questions, same truncation across all three conditions.

The mechanism is real — truncation drops the middle of the document, while Lingua’s extractive classifier keeps answer-bearing tokens distributed across it. But n=15 is directional only: Fisher’s exact two-sided p ≈ 0.18, so this table does not establish the effect. Treat the N=503 sweep above as the load-bearing result. Per-question records: docs/blog/data/lbv2-2026-05-03/.

Internal benchmark — coding-agent transcript

A realistic 9-message agent transcript — user question, file reads, grep, log dumps, failed edit, error trace — totaling ~2.1K tokens. Run through leanctx.Anthropic with content-aware routing (code → verbatim, errors → verbatim, prose → Lingua):

What got preserved verbatim (asserted programmatically by the agent-structural bench scenario):

• A 2 KB Python source file inside a tool_result block — byte-identical
• A Python traceback in an is_error tool result — byte-identical
• Every tool_use_id and the name / input of every tool_use block — tool linkage and tool-call payloads untouched
• edit_file’s new_str argument — the actual code edit isn’t rewritten

What actually compressed:

• A 3.4 KB log dump shrank to 1.9 KB (45 % reduction) — the legitimate compression target
• Grep results and prose reasoning blocks shrank by 30–50 %

Reproduce: leanctx bench run agent-structural --workload agent — runs the real LLMLingua-2 model, ~30 s on Apple Silicon, no API key required. Status flips to failure with named invariants if any regress; CI-gateable.

SelfLLM cross-provider comparison

Same 1.7 KB SRE-incident document through SelfLLM against each provider’s cheapest tier:

All three preserved every timestamp, metric value, and action item with no hallucination. Combined with Lingua (LLMLingua-2 local) hitting 44.7 % char reduction on the same document at zero marginal cost, leanctx covers the full speed/cost/quality trade-off space.

Full methodology, per-provider output samples, cost analysis, bugs found in flight: docs/benchmarks/.

Integrations

Deployable integrations against third-party stacks, each with a working sidecar, a connector, and a measurement harness.

Integrations are opt-in and fail-open by construction: if the sidecar is unreachable, slow, or returns anything that fails the invariant check, the original uncompressed request goes upstream. A compression outage costs savings, never availability.

Running leanctx against your stack? Open an issue — measured results get listed here.

How it works

Invariants

Checked after recompose, before the request goes upstream:

• message count and ordering match the input
• every tool_use_id, tool name and tool input unchanged
• code and stack traces inside tool results byte-identical
• compression path responded within its timeout

Any failure sends the original uncompressed request. Fails open: a compression outage costs savings, never availability.

The pipeline

leanctx wraps your existing SDK call and applies a configurable compression pipeline before the request hits the wire.

your code
   ↓
leanctx.Anthropic / OpenAI / Gemini    ← drop-in wrapper
   ↓
Middleware (mode=on/off, threshold)
   ↓
Per-message pipeline:
   classify (code | error | prose | …)
        ↓
   route to compressor:
        Verbatim  — never touch (code, errors, tool calls)
        Lingua    — LLMLingua-2 local, free marginal cost
        SelfLLM   — your configured LLM (Anthropic/OpenAI/Gemini), highest quality
   ↓
real Anthropic / OpenAI / Gemini SDK → API

Two layers of config:

• mode — "on" to compress, "off" to passthrough. Off is safe to leave deployed.
• routing — maps content types (code / error / prose / unknown / long_important) to compressors (verbatim / lingua / selfllm).

A fully-loaded production config:

from leanctx import OpenAI

client = OpenAI(leanctx_config={
    "mode": "on",
    "trigger": {"threshold_tokens": 2000},  # don't bother below this
    "routing": {
        "code":           "verbatim",   # never touch code
        "error":          "verbatim",   # never touch stack traces
        "prose":          "lingua",     # local LLMLingua-2
        "long_important": "selfllm",    # cheap LLM summarization
    },
    "lingua":  {"ratio": 0.5, "device": "cpu"},
    "selfllm": {"model": "gpt-4o-mini", "api_key": "sk-...", "ratio": 0.3},
    "observability": {"otel": True},     # opt-in OpenTelemetry
})

Compose with provider caching

leanctx is complementary to Anthropic / OpenAI / Gemini prompt caching, not competitive:

• Provider caching wins on stable prefixes: system prompts, tool definitions, retrieved-document pools that don’t change across requests. Up to 90 % discount on cached reads.
• leanctx wins on dynamic per-query content: chat history, freshly retrieved docs, tool outputs, log dumps that vary every call.
• They compose. Mark your stable prefix with cache_control (provider-specific) and let leanctx compress the variable suffix. Both savings stack.

The OTel telemetry leanctx emits includes a provider label that you can correlate with provider-side cache-hit metrics in the same dashboard.

Observability (v0.3)

leanctx emits OpenTelemetry spans + metrics for every compression call, opt-in via leanctx_config["observability"]["otel"]. The library is API-only: it never owns the OTel SDK or registers providers. The application configures OTel; leanctx emits.

client = leanctx.Anthropic(
    leanctx_config={
        "mode": "on",
        "observability": {"otel": True},
    },
)

Each wrapper-routed call produces one root leanctx.compress span (provider, method, input_tokens, output_tokens, cost_usd, duration_ms) plus per-compressor child spans. Five metrics — 4 counters + 1 histogram — labeled by provider/method/status. Closed leanctx.method taxonomy: passthrough | below-threshold | empty | opaque-bailout | verbatim | lingua | selfllm | hybrid.

See docs/observability.md for the full attribute reference, stream-lifetime contract, app-side OTel SDK setup, and cardinality guidance.

Reproducible benchmarks (v0.3)

The leanctx bench CLI ships seven named scenarios with versioned JSON output (schema_version: "1"):

leanctx bench list                                  # show registered scenarios
leanctx bench run lingua-local --workload rag       # offline, no API key
leanctx bench run agent-structural --workload agent # 5 invariants enforced
leanctx bench run anthropic-e2e --workload chat     # full stack, respx-mocked
leanctx bench run selfllm-anthropic --workload rag  # live API, set ANTHROPIC_API_KEY
leanctx bench run longbench-v2 --workload rag       # public LongBench v2 ablation

Versioned schema, multi-run isolation (--runs N constructs fresh client/middleware each run), clean diagnostics for missing extras / API keys (exit 3, no traceback). Built so downstream tooling can consume the JSON without breaking on schema changes.

Install

pip install leanctx                              # core (passthrough only — useful for testing the wrapper)
pip install 'leanctx[anthropic,openai,gemini]'   # provider SDKs
pip install 'leanctx[lingua]'                    # + LLMLingua-2 local compression (~1.2 GB on first call)
pip install 'leanctx[otel]'                      # + OpenTelemetry API/SDK
pip install 'leanctx[bench]'                     # + respx for offline scenarios
pip install 'leanctx[longbench]'                 # + HuggingFace datasets for LongBench v2
pip install 'leanctx[server]'                    # + FastAPI/uvicorn HTTP compression sidecar
pip install 'leanctx[all]'                       # everything

HTTP sidecar (for non-Python callers)

Run leanctx as a long-lived HTTP service so a TypeScript proxy, a LiteLLM callback, a Go gateway — any non-Python stack — can use leanctx compression over POST /compress:

pip install 'leanctx[server,lingua]'
leanctx-serve --port 8459
curl -s localhost:8459/compress -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"<long prose>"}]}'

It compresses system/user/assistant prose and forwards tool results + multimodal content verbatim, preserving message order/count. See docs/server.md.

Docker:

docker build -t leanctx:slim .                             # 341 MB, all provider SDKs
docker build -t leanctx:lingua --build-arg LINGUA=true .   # + LLMLingua-2, ~3 GB

Supported providers

* Gemini text-only requests compress fully. Requests that include function_call, function_response, or multimodal (inline_data) parts automatically bail out to passthrough — leanctx never rewrites tool-call payloads (would change tool semantics) and doesn’t touch images. Multimodal + function-call compression is on the v0.3.x roadmap. Spans for these calls carry leanctx.method = opaque-bailout so you can monitor the share.

12 wrapper request paths instrumented (sync + async × stream + non-stream × 3 providers). Stream-path span lifetime closes at the first of: iterator exhaustion, explicit .close(), or __del__ GC backstop — duration_ms covers the full stream lifetime.

Status

v0.3.1 is on PyPI. Built across a 5-round Codex-reviewed RLCR loop; 257 tests passing, ruff + mypy --strict clean across 40 source files.

Roadmap

• v0.1 — Python SDK, drop-in wrappers, LLMLingua-2 + SelfLLM (Anthropic), classifier, router, dedup + purge-errors strategies, LangChain helpers, Docker
• v0.2 — SelfLLM on OpenAI + Gemini, block-aware compression (tool_use / tool_result preserved), Gemini contents normalization, LCEL compress_runnable
• v0.3 — OpenTelemetry observability across 12 wrapper paths, leanctx bench CLI (6 scenarios + versioned schema), agent-structural invariant enforcement, public release v0.3.1 — 2026-04-26
• Full 503-item LongBench v2 sweep — externally run and audited, methodology corrected, published — 2026-06-13; per-item records committed and independently re-run
• v0.3.x — ghcr.io Docker publish, OpenAI Responses-API intercept, multimodal + function-call compression for Gemini, LlamaIndex helpers, TypeScript SDK compression port
• v0.4 — per-tenant attribution (with cardinality cap), Helm chart / K8s sidecar, stateful session dedup with explicit session IDs

License

MIT. See LICENSE.