sndr_core_engine

SNDR Core Engine

Genesis vLLM Patches — runtime vLLM patches that run frontier-class open LLMs — Qwen3.6 (7B · 27B · 35B-A3B) and Gemma 4 (26B · 31B · DiffusionGemma) — on consumer NVIDIA GPUs with 24 GB (RTX 3090 / 4090 / 5090, RTX A5000 / A6000): ~1.5× faster inference, quantized tool calling that works, MTP speculative decoding, and up to 280K-token context — no fork, no rebuild.

Built on vLLM, so it serves the models the engine supports; the deep optimizations (TurboQuant KV, hybrid GDN, spec-decode, tuned kernels) are family-tuned for Qwen3.6 and Gemma 4 — the two families we validate on every pin.

🎮 Own a different card? The 24 GiB envelope is class-wide, and sndr up auto-projects VRAM for your GPU. RTX 4090 · RTX 5090 (32 GiB) · dual RTX 3090 — honest per-class gotchas (Ampere-calibrated tuning, no-NVLink P2P, idle-VRAM headroom) in the FAQ.

Contents: Get running · Who is this for · Why SNDR Core · How it compares · What it is · How it works · The platform end-to-end · Headline numbers · Fleet validation · Persistent memory · Pick your path · Install & run · FAQ · Documentation map · Repository structure · Contributing

Turn a consumer NVIDIA card into a production local-AI server. SNDR Core transforms the open-source vLLM engine in memory at boot — no fork, no rebuild — so frontier-class open models (Qwen3.6 up to 35B-A3B, Gemma 4 up to 31B) run ~1.5× faster than stock vLLM with up to a 280K-token served context, on hardware you can actually buy (A5000, RTX 4090 / 5090, A6000 — and yes, the 3090). One paste installs it; a real GUI Control Center drives it.

Two products, one engine: ⚙️ the runtime vLLM patch-overlay (faster inference) + 🧠 a persistent neural-graph memory that makes every model — local and cloud — remember and get smarter over time. Apache-2.0, self-hosted, fully auditable. 329 patches across ~23 families.

Sound familiar?

• You want 70B-class quality but only have 24 GB of VRAM.
• Tool calls break the moment you quantize the model.
• vLLM OOMs on consumer GPUs the moment you ask for long context.

That is exactly the gap this project closes — measured, reproducible, on hardware you already own.

Get running — two commands

curl -sSL https://raw.githubusercontent.com/Sandermage/sndr_core_engine/main/install.sh | bash
sndr up          # auto-picks a preset for your GPU → downloads the model → launches → opens the GUI

That’s it — sndr up detects your rig, downloads the weights (skipped if present), starts the engine and the Control Center, and opens your browser at the Control Center (http://127.0.0.1:8765). Prefer the terminal? sndr run does the same and drops you straight into a chat prompt. New here? Start with docs/GETTING_STARTED.md.

The engine needs Linux + CUDA + Docker. On a Mac or Windows laptop you can’t run the engine locally — but you can drive a Linux rig remotely with the same sndr CLI and GUI. See docs/RUN_ON_MAC.md (Mac), docs/RUN_ON_WINDOWS_WSL.md (Windows), docs/RUN_ON_LINUX.md (full local stack), and docs/REMOTE_ENGINE.md (client-mode reference).

Who is this for

• Homelab operators — you own (or can buy) a 24 GB-class NVIDIA card and want frontier-class local inference without datacenter hardware. Start: docs/SINGLE_CARD.md.
• On-prem / privacy deployments — data can’t leave the building. Self-hosted, Apache-2.0, every applied patch logged; nothing phones home.
• Agent builders — you need an OpenAI-compatible endpoint with tool calling that survives quantization, long agentic tool-chains, and a persistent memory gateway. Start: docs/MCP.md + docs/memory/MANUAL.md.
• Researchers / performance engineers — a 329-entry patch registry with per-patch evidence, a bench suite with CV methodology, and reproducible numbers. Start: docs/BENCHMARKS.md + docs/PATCHES.md.

🚀 New here? → docs/GETTING_STARTED.md — who it’s for, what you get, and the one install line. ❓ Quick answers → docs/FAQ.md — the questions everyone asks first. 🧠 New to local AI? → docs/LOCAL_AI_PRIMER.md — GPUs, engines, MoE, and quants in plain English. 📖 Hit an unfamiliar term (TPS · KV · MTP · TurboQuant · GDN)? → docs/GLOSSARY.md. 💸 Self-host or cloud? → docs/COMPARISONS.md — the cost-crossover trade.

Why SNDR Core — what you get

How it compares

Honest snapshot: the SNDR and stock-vLLM cells are measured on our reference rig; the rest are qualitative — we have not benched Ollama / llama.cpp / TGI here (the repo ships a llamacpp-qwen3.6-27b-q4km-1x preset if you want a measured llama.cpp row on your own rig).

The longer self-host vs cloud (and engine-alternative) discussion lives in docs/COMPARISONS.md.

What it is

A drop-in runtime patcher for vLLM. It pins to a specific vLLM nightly commit and applies 329 small, surgical changes — text edits at known anchors, class-rebind wrappers, and FastAPI middleware — that together turn an out-of-the-box vLLM into a production-grade Qwen3.6 / Gemma 4 inference server on consumer NVIDIA hardware (A5000, RTX 4090 / 5090, A6000, 3090, …) where vLLM upstream mostly targets datacenter SKUs.

It is not a fork of vLLM, a quantizer, a new inference engine, or a training framework. Patches retire automatically when upstream merges the underlying fix.

How it works

The overlay / apply model. Genesis never edits vLLM on disk. At every process start the plugin registers via vLLM’s vllm.general_plugins entry point (loaded in the main process, the engine, and every worker rank) and the dispatcher walks PATCH_REGISTRY. Each patch declares an applies_to version range and an apply method — a text edit at a unique source anchor, a class-rebind wrapper, or FastAPI middleware. Patches whose anchors match and whose range covers the live pin apply; the rest print [SKIP — applies_to mismatch] and no-op. The result is an in-memory overlay: the same wheel, transformed at boot, with a structured apply summary (applied=N skipped=M failed=0) and an audit trail. Nothing is written to the vLLM package tree.

Patch families. The 329 entries group into ~23 canonical families. The largest: attention.turboquant (k8v4 KV-cache quant), spec_decode (MTP / ngram speculative decoding), attention.gdn (hybrid Gated-DeltaNet linear attention), gemma4 (Gemma-4 enablement), kv_cache, compile_safety, worker, serving, tool_parsing, and moe. The full table is docs/PATCHES.md (curated) + docs/PATCHES_AUTO.md (generated from the registry).

Pin lifecycle. Genesis pins to one canonical vLLM nightly at a time, plus an optional previous pin held for rollback during validation — at most two (“≤2-pin policy”). A bump happens only on an explicit instruction naming the target pin; there are no proactive pulls. The candidate is validated before promotion (anchor-drift resolved, the bump-preflight gate clean, boot-smoke + tokenizer-fingerprint + canonical bench), then the old 2-back pin is dropped. Current: dev748 (2dfaae752); rollback: dev714 (09663abde). See docs/PIN_BUMP_PLAYBOOK.md (canonical) + docs/ANCHOR_SOT.md.

Model catalog (current registry).

Per-model deep-dives + the V2 layered config system: docs/MODELS.md. Hardware envelope: docs/HARDWARE.md.

Launching. Boot any model through a preset — the launcher resolves the preset, runs preflight, and renders the docker run (or podman / bare-metal / k8s) command for you with the correct pin, mounts, and env:

sndr launch prod-qwen3.6-35b-balanced            # boot a preset
sndr launch prod-qwen3.6-35b-balanced --dry-run  # inspect the rendered command, no boot

Note: prod-qwen3.6-35b-balanced is the shipped K=3 balanced default — and the right pick for a single user at a keyboard (latency-tuned, max_num_seqs=2). It is what the zero-decision sndr up / sndr quickstart auto-picks for a lone-user rig. Reach for prod-qwen3.6-35b-multiconc only when serving many concurrent requests — it is throughput-tuned (max_num_seqs=8, ~672 t/s aggregate) and trades single-stream latency for that aggregate. The headline numbers below come from the live PROD stack at MTP K=5 (re-tuned 2026-06-19, +15.8 % single-stream vs K=3) — expect the K=3 preset to land correspondingly below the K=5 figures.

Full operator manual: docs/USAGE.md.

What the platform does end-to-end

Beyond “faster tokens”, SNDR Core is a full operations platform around the patched engine — every layer below is shipping today and exercised by the CI gates:

Headline numbers (v12.1.0 current registry)

Reference rig: 2× RTX A5000 24 GB (Ampere SM 8.6), driver 580.142, CUDA 13.0.2, TurboQuant k8v4, TP=2. Live PROD stack: Qwen3.6-35B-A3B (AWQ checkpoint), MTP K=5, qwen3_xml tool parser, 280K served context.

Fresh canonical bench — pin dev748 promotion gate, 2026-07-04:

Same-day reference — pin dev714, 2026-07-04, extended canonical suite (kept as the labeled comparison run the parity verdict above is measured against):

Earlier measured records, each labeled with its pin:

280K served context verified on the PROD preset (max_model_len: 280000), with linear decode scaling through 32K in the fresh suite. Full methodology, historical comparisons, and per-rig reproduction recipes: docs/BENCHMARKS.md.

Current pin: vLLM 0.23.1rc1.dev748+g2dfaae752 (commit 2dfaae752, image vllm/vllm-openai:nightly-2dfaae752). Per the ≤2-pin policy, dev714 (0.23.1rc1.dev714+g09663abde, commit 09663abde) is retained as the rollback pin; dev672 is dropped. A stable track also exists: the registry recognizes the tagged release v0.24.0 for operators who prefer release pins over nightlies. sndr/pins.yaml is the single source of truth for all three. dev748 was promoted 2026-07-04 through the full playbook chain — preflight re-anchor → boot gate (apply failed=0 across the whole 7-model fleet; four lanes initially booted the dev714 rollback engine via a stale image_digest and were re-run on verified dev748 on 2026-07-05 — see the fleet table below) → bench gate (242.5 t/s wall — parity within CV vs the same-day dev714 run, no regression; tool-call 7/7) → receipts → tag rotation — see docs/PIN_BUMP_PLAYBOOK.md (canonical) and docs/ANCHOR_SOT.md. The per-model table below is the historical dev148 K-tune cycle, kept for cross-model context and labeled with its pin; the fresh dev748 headline above supersedes it for the 35B PROD stack.

Validated across the fleet — 7 models (dev748; 2026-07-04 window + 2026-07-05 re-run)

The works-everywhere proof the project leans on: during the dev748 promotion window every launchable model in the catalog was booted sequentially (2× RTX A5000, TP=2), smoke-tested and mini-benched — and all seven applied their patch sets with failed=0. Post-release audit (2026-07-05): four lanes had initially booted the dev714 rollback engine (a stale hardware image_digest beat the dev748 tag at render; digest + gate since fixed) — those four were re-run on verified dev748 on 2026-07-05 (per-lane in-container version + bench fingerprint checks), and the table shows the re-run numbers. Accept rates are bench-window rates. Condensed from the full sweep table (with the labeled dev714 first-pass rows) in docs/BENCHMARKS.md:

(27B thinking mode loops — a known pre-existing model trait; chat is validated with enable_thinking:false and the tool-agent workload is unaffected. Details + footnotes in docs/BENCHMARKS.md.)

Recent battle-validations. The PN520 story is the class every operator recognizes: the INT4 27B booted clean — patches applied, server healthy — and then produced garbage output. Root cause was an upstream GDN loader change silently dropping the checkpoint’s split BF16 shards from the fused in_proj_ba parameter, leaving the linear-attention layers uninitialised; the PN520 loader revert routes all 96 in_proj_ba shards correctly, and the degeneration is cured (coherent chat + tool calls in the sweep above). In the same window, P100 (FlashInfer FULL-CG spec-decode) was runtime-validated on the fp8kv lane — coherent generation, 0 errors — and PN351’s dev748 anchor variant was battle-validated on the head_dim=512 Gemma 4 31B in the 2026-07-05 re-run: the applied variant was read back from the live dev748 container file, and the lane served chat + 7/7 tool-calls with window accept 0.744.

Validated rig baseline — 2026-06-19 (measured on pin 0.23.1rc1.dev148+gb4c80ec0f)

Full model-cycle re-test on the reference 2× A5000 rig after the MTP K=3→K=5 re-tune, recorded on pin dev148 with the FP8 35B checkpoint of that cycle (the live PROD stack has since moved to the AWQ checkpoint — fresh dev748 numbers in the headline table above). The pin has since bumped dev148 → dev301 → dev424 → dev672 → dev714 → dev748 (current) with no decode regression (anchor regen confirmed at each bump). Each model boots the Genesis apply pipeline, applies its patch set, and is benchmarked / smoke-tested live (tools/genesis_bench_suite.py, single-stream warm sweep). The 35B and 27B single-stream rows are the dev148 K=5 re-tune record; Gemma stays K=3 (its separate drafter is optimal at K=3). Note: the live 27B config has since moved to MTP K=4 — the max coherent K for its INT4 tool-calls (K=5 emitted unparseable tool-call tokens on dev714); K=4 warm decode is ~125 t/s, within CV of the K=5 record below.

The 35B and 27B clear their historical peak band — the K=5 re-tune lifts single-stream decode to 239.7 / 127.4 t/s (+15.8 % / +8.2 % vs K=3) within CV → the v12 platform carries no decode regression. PN-FP8MOE-KPAD (backport of open vLLM PR #45703, model-agnostic Marlin-MoE intermediate-pad) plus G4_26 (backport of #45774, DiffusionGemma TP>1 vocab-sharded soft-embed all-gather) make DiffusionGemma the first block-diffusion FP8-MoE checkpoint to boot AND serve coherently at TP=2 on consumer Ampere without a kernel rebuild — validated 2026-06-17 (clears the Marlin N=352 thread-tile crash, then the probs @ embed_weight [131072,2816] TP-vocab shape mismatch; the coherent generation confirms the soft-embed all-gather yields correct TP=2 output).

Persistent memory — neural-graph (new in v12)

A brain-like persistent memory that makes every model — the internal vLLM engines and external models behind your proxy — smarter over time. Knowledge is stored as a graph whose nodes auto-form connections and cluster into “clouds” (like Obsidian), is recalled by vector similarity plus spreading activation across the graph, and decays / reinforces like human memory. It ships as one CPU-only container (Postgres + pgvector + API + GUI + gateway) — the GPU engines are untouched.

By the numbers (v12, all verified): 2 storage backends (in-memory + Postgres/ pgvector) proven identical in CI · real CPU embedder (Model2Vec) semantic match 0.85 related vs 0.01 unrelated · ~100 unit tests + a leak-soak, run on both backends (Postgres against a live pgvector in CI) · one container · zero GPU on the hot path.

GUI — Memory panel (Control Center → Engine → 🧠 Memory; served same-origin). Real screenshot of the live Control Center (dark theme):

One container, one docker run, and any OpenAI client pointed at the gateway gains memory. Deployment recipe, brain mechanics + tuned constants, every endpoint, config, security, and troubleshooting: docs/memory/MANUAL.md.

Pick your path

I want to… (by machine):

Install & run

# 1. install — detects OS / Python / GPU / vLLM, installs the plugin + `sndr` CLI
curl -sSL https://raw.githubusercontent.com/Sandermage/sndr_core_engine/main/install.sh | bash

# 2. run — auto-picks a preset for your GPU, downloads the model, launches, opens the GUI
sndr up            # …or `sndr run` for a terminal chat prompt instead of the GUI

sndr up and sndr run both download the model if it isn’t already present (skipped when it is), so step 2 is genuinely one command. Want to see the plan first? Add --dry-run. Pick a named preset with sndr up <preset> (browse them with sndr preset list).

Five-minute walk-through + Day-1 acceptance: docs/QUICKSTART.md. A different vLLM pin, workload, or non-interactive flag set: docs/INSTALL.md.

FAQ

The questions people actually search for. Longer answers (and ~25 more questions) live in docs/FAQ.md.

Can I run a 35B-class model on 24 GB of VRAM? On a single 24 GB card the validated recipe is the 27B INT4 preset (qa-qwen3.6-27b-tq-1x, 78K context); the 35B MoE needs 2× 24 GB at TP=2, where the PROD stack decodes at 242.5 t/s (pin dev748, 2026-07-04). sndr preflight and sndr kv-calc tell you what fits before you download weights. → docs/SINGLE_CARD.md · docs/HARDWARE.md

Why do tool calls break on quantized models? Quantization amplifies upstream parser fragility — <think> tags, multi-tool prompts, and streaming chunk splits produce malformed calls on stock configs. Genesis ships a dedicated tool-call patch family (P59 / P61 / P62 / P64 / P68 / P69) around the native qwen3_xml streaming parser: 7/7 PASS on the dev748 promotion gate and 8/8 on the extended battery (dev714, both 2026-07-04). → docs/FAQ.md · docs/TROUBLESHOOTING.md

How much faster is speculative decoding? The MTP K=3→K=5 re-tune alone lifted 35B single-stream decode +15.8 % (207 → 239.7 t/s, pin dev148, 2026-06-19); the full recommended patch set is ≈1.5× stock vLLM on the same commit (+53 % on 35B, +46 % on 27B; dev148, 2026-06-19). Current canonical figure: 242.5 wall TPS (dev748, 2026-07-04). → docs/SPEC_DECODE_GUIDE.md · docs/BENCHMARKS.md

Is this a fork of vLLM? No. It runs against an unmodified pinned vLLM wheel and applies patches in memory at boot; toggle Genesis on/off with env flags on the same binary. Each patch declares an applies_to version range and retires automatically when upstream merges the underlying fix. → docs/FAQ.md

How does 280K context fit on 24 GB cards? TurboQuant k8v4 KV-cache quantization (8-bit keys, 4-bit values) frees 2–4× more concurrent KV slots, which is what lets the PROD preset serve max_model_len: 280000 — above the model’s published 256K limit — with linear decode scaling through 32K (LINEAR_OK, dev748, 2026-07-04). sndr kv-calc projects the exact KV bytes for your card. → docs/KV_PROJECTOR.md · docs/BENCHMARKS.md

Can I run it without Docker? Yes — it is a regular Python package that patches a vLLM installed in the same environment; sndr model-config render <key> --runtime bare_metal emits a venv launch script. Kubernetes and Proxmox lifecycles are wired via python3 -m sndr.cli.legacy service install <key>. → docs/FAQ.md

Is it free? Everything in this repo — sndr/**, tests, docs, bench data — is Apache 2.0. The license gate in the tree guards a commercial overlay that is absent from the public tree; it does not restrict the community tier. → docs/LICENSE_POLICY.md

Documentation map

Full docs index: docs/README.md.

Repository structure

The layout separates the shippable engine from the maintainer tooling and vendored third-party code, so the published wheel stays small and the apply pipeline stays auditable.

Contributing

If this saves you a GPU upgrade, a ⭐ helps others find it.

Bug reports, new patches with empirical evidence, new model recipes, and cross-rig bench reports are all welcome. The full workflow (anchor conventions, lifecycle ratchet, pin-bump playbook, PR template) is in docs/CONTRIBUTING.md. Security disclosures go through SECURITY.md.

Ecosystem / Related

• vLLM — the upstream engine SNDR Core patches. Genesis is an overlay, not a fork; each patch retires as upstream merges the underlying fix.
• Hugging Face — where the model weights the presets pull come from.

Credits + license

Apache-2.0 (see LICENSE). Per-patch attribution and upstream PR linkage in docs/CREDITS.md.

Author: Sandermage (Aleksandr Barzov), Odessa, Ukraine. Sponsorship channels (voluntary, no obligations) and hardware-loan contact: docs/SPONSORS.md.