atlas

Pure Rust LLM inference, from the device in your hand to the datacenter rack.

Website · Docs · Blog · Discord · Deployment Guide

Atlas is an open-source LLM inference engine written in pure Rust and CUDA. It serves an OpenAI-compatible server (plus Anthropic and Responses APIs) from a single ~75 MB binary — no Python, no PyTorch, no runtime compilation — with hand-tuned CUDA kernels per (hardware × model × quantization) target, NVFP4 and FP8 quantization, speculative decoding (MTP draft heads and DFlash block diffusion), radix-tree prefix caching, and expert parallelism across nodes. It is verified today on the NVIDIA DGX Spark (GB10, Blackwell SM121), compiles the same CUDA source for AMD Strix Halo (gfx1151) through SCALE, and on the published GB10 concurrency ladder it out-serves vLLM at every rung from C=1 to C=128 — conditions below.

Receipts, not adjectives:

• Our fused Qwen Gated DeltaNet kernel is merged into Hugging Face Transformers.
• We sit on the MLCommons Edge-LLM taskforce and helped shape the MLPerf Inference v6.1 edge agentic benchmark; our v6.1 submission is in (closed edge division, GB10 and gfx1151 from the same CUDA source), with results under embargo until MLCommons publishes.
• Every release image passes a serve gate: boot, coherence, tool calls, and throughput within tolerance of a committed baseline. A release that ships slower than its baseline fails the gate.
• The engineering story is written up in the open on the Atlas blog, starting with the seven tenets behind the engine.

📑 Table of Contents

• 🚀 Quick Start
• 🖥️ Supported Hardware
• 📦 Supported Models
• ⚡ Performance
• 🗜️ KV Cache Quantization
• 🏛️ Architecture
• 🧭 Why Atlas Exists
• 🔌 Adding a New Hardware Target
• 🧬 Adding a New Model
• 🔬 Kernel Debugging
• 🔐 How a Change Lands
• 🤝 Community and Contributing
• 📚 Citations
• ⚖️ License and Enterprise Edition

🚀 Quick Start

One command

curl -fsSL https://atlascybernetics.ai/install.sh | sh
atlasctl run qwen3.6-35b-a3b-fp8-mtp

The script downloads a prebuilt atlasctl, verifies its checksum, and installs it to ~/.local/bin — no Python, no Rust toolchain. Prefer not to pipe curl into a shell? cargo install atlasctl does the same from source. Every runnable model maps to a recipe in atlas-recipes, so the catalogue cannot list a model we do not ship.

Docker

The whole supported model matrix lives in one image. Pull it, mount your HuggingFace cache, and point serve at any model ID from the model table.

Recipe 0 — no flags, pick a model in the TUI

Omit the model ID and serve boots into the Library — pick a model and recipe interactively (TTY only):

docker run -it --rm --network host --gpus all --ipc=host \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  -v "${HOME}/.avarok:/root/.avarok" \
  avarok/atlas-gb10:latest serve

• -it — the TUI needs a real terminal to render (and Esc to quit).
• --rm — throwaway container; nothing to clean up after the session.
• --network host — the served port is reachable on localhost directly, no -p mapping.
• --gpus all — hands the GB10 to the container.
• --ipc=host — host-sized shared memory; the Docker default 64 MB /dev/shm is too small for CUDA.
• -v ~/.cache/huggingface — reuse the host’s model cache instead of re-downloading weights.
• -v ~/.avarok — persist Atlas state (recipes, benchmark records, artifacts) across runs.

Recipe A — Qwen3.6-35B-A3B (FP8 hybrid MoE, the daily driver)

35 B params, 3 B active, GDN + attention + 256-expert MoE, MRoPE-positioned vision tower (text-only here).

docker pull avarok/atlas-gb10:latest

sudo docker run -d --name atlas \
  --network host --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  avarok/atlas-gb10:latest \
  serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8888 \
    --max-seq-len 65536 \
    --kv-cache-dtype fp8 \
    --kv-high-precision-layers auto \
    --gpu-memory-utilization 0.90 \
    --scheduling-policy slai \
    --enable-prefix-caching \
    --speculative \
    --num-drafts 2 \
    --tool-call-parser qwen3_coder

Why these flags:

• --max-seq-len 65536 — 64K window for long agent traces, file reads, multi-step tool use.
• --kv-cache-dtype fp8 --kv-high-precision-layers auto — half the memory of BF16, no measurable quality loss; the boundary attention blocks stay BF16, where the routing distribution is most sensitive. auto is not a heuristic — it is a fixed alias for 2 (serve_phases/kv_cache.rs), alongside max/all meaning “every attention layer”. Because it is non-zero it also suppresses the per-dtype automatic promotion that 0 would trigger under a turbo* KV dtype.
• --scheduling-policy slai — SLAi scheduler. Not the default — serve defaults to fifo, so this flag has to be passed to get SLO-aware ordering. It reorders concurrent sequences to keep MTP verify batches dense and prefills shortest-prompt-first.
• --enable-prefix-caching — radix-tree prefix cache; tool-use sessions reuse the system prompt + tool-defs + earlier turns.
• --speculative --num-drafts 2 — MTP draft head proposes 2 tokens per step. No --mtp-quantization flag ⇒ defaults to BF16, which gives the highest acceptance rate (lossier MTP projections lower acceptance and usually worsen end-to-end tok/s, despite the faster draft forward).
• --tool-call-parser qwen3_coder — explicit Qwen XML tool format. Atlas auto-resolves the right parser from tool_defaults.toml per model; pass it anyway in production scripts.

Recipe B — Qwen3.5-35B-A3B (NVFP4, ~131 tok/s with MTP K=2)

The fastest model in the matrix on a single Spark.

sudo docker run -d --name atlas \
  --network host --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  avarok/atlas-gb10:latest \
  serve Sehyo/Qwen3.5-35B-A3B-NVFP4 \
    --port 8888 \
    --max-seq-len 65536 \
    --kv-cache-dtype fp8 \
    --kv-high-precision-layers auto \
    --gpu-memory-utilization 0.90 \
    --scheduling-policy slai \
    --enable-prefix-caching \
    --speculative \
    --tool-call-parser qwen3_coder

--num-drafts is omitted so it defaults to 1, i.e. MTP K=2 (the CLI defines --num-drafts 1 as K=2, 2 as K=3). K=2 is the measured-fastest verify width for this model; K=3 is slower.

Recipe C — Qwen3.5-122B-A10B (NVFP4, single Spark)

The 122B NVFP4 weights + Atlas runtime overhead leave only ~2 GB for KV cache on a 119.7 GB GB10, so this recipe sacrifices --speculative (the MTP draft head + draft KV costs ~1.5 GB) to keep a real 16 K context window. Verified end-to-end: model loads, /v1/chat/completions answers correctly, 4-way concurrent serves cleanly.

sudo docker run -d --name atlas \
  --network host --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  avarok/atlas-gb10:latest \
  serve Sehyo/Qwen3.5-122B-A10B-NVFP4 \
    --port 8888 \
    --max-seq-len 16384 \
    --kv-cache-dtype fp8 \
    --kv-high-precision-layers auto \
    --gpu-memory-utilization 0.92 \
    --scheduling-policy slai \
    --max-batch-size 1 \
    --max-num-seqs 4 \
    --oom-guard-mb 1024 \
    --ssm-cache-slots 0 \
    --tool-call-parser qwen3_coder

For 122B with both --speculative and a 64 K window, move to EP=2 across two Sparks (QUICKSTART.md §7). For long contexts on a single Spark, add --high-speed-swap --high-speed-swap-dir /path/on/nvme --high-speed-swap-cache-blocks-per-seq 64 — HSS keeps a rolling 1024-token KV window in HBM and streams older blocks to NVMe through an io_uring orchestrator. The container needs --security-opt seccomp=unconfined --ulimit memlock=-1 for io_uring access.

Hitting the endpoint

Atlas speaks the OpenAI, Anthropic, and Responses APIs on the same port. curl, the OpenAI SDK, Open WebUI, opencode, Cline, Claude Code — point them at port 8888:

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model":"atlas",
    "messages":[{"role":"user","content":"Hello!"}],
    "max_tokens":256
  }'

Per-model recipes (vision input, video input, multi-node EP=2, single-GPU 122B with the tighter budget) live in QUICKSTART.md, and the long-form manual is at docs.atlascybernetics.ai.

🖥️ Supported Hardware

One engine, one kernel tree per target, no generic fallbacks. Each directory under kernels/ is a hardware target with its own HARDWARE.toml:

Porting to new silicon is a scoped piece of work, not an architectural change — see Adding a New Hardware Target.

📦 Supported Models

Every supported model runs off one multi-model binary; the right kernel set is selected at startup from the model’s config.json. No swapping images, no rebuilding, no per-model magic — just point Atlas at a HuggingFace ID.

The kernels/gb10/ tree carries additional targets in various stages of bring-up (Qwen3.8, DeepSeek-V4-Flash, LongCat-Flash-Lite, and more); the GB10 Deployment & Compatibility Guide is the authoritative model × quant matrix, kept current as targets graduate.

This is a starting point, not a destination. The plug-and-play design below exists precisely so that AMD, Apple Silicon, Intel, and the next round of Blackwell parts can land here as community contributions, and so that next quarter’s model families slot in the same way this quarter’s Qwens did. We did the hard part — bolting in the abstractions while bringing up the first wave of targets — so that adding the next one is a weekend, not a quarter.

⚡ Performance

We are not going to spend much real estate on benchmark theatre. Every number below carries its conditions, and every harness that produced one is in this repository. If you reproduce a faster competing number, file an issue — we would rather be measured than congratulated.

Atlas vs vLLM under concurrency (the number that matters)

Agentic work does not arrive one conversation at a time. It arrives as fleets of tool-calling agents, and the engine underneath is judged where requests pile up. On the published concurrency ladder Atlas out-serves vLLM at every rung from C=1 to C=128, against whichever vLLM configuration is faster at that rung:

Conditions: unsloth/Qwen3.8-27B-NVFP4 (dense 27B hybrid, 48 GDN + 16 attention layers), Atlas 1.0.0-beta-preview vs vLLM 0.27.1, same GB10 box, same checkpoint, same client, back-to-back legs. ISL 128 / OSL 1024, temperature 0, seed 42, thinking disabled on both engines, presence/frequency penalties pinned to 0.0 on both, mean aggregate tok/s over 3 timed reps with 1 warmup discarded. The margin is widest at the top because between C=64 and C=128 Atlas keeps scaling while vLLM’s faster mid-ladder configuration (MTP) falls below its own C=64. The full campaign log — including the rungs we lost along the way — is in bench/ladder38/RESULTS.md.

Single-stream throughput on one GB10

The numbers below are what the binary in this repository does on a single NVIDIA GB10, on a short prompt ("What is the capital of France?", max_tokens ≤ 30, temperature = 0.1), measured end-to-end through the HTTP API. scripts/sweep_all_models.sh is the harness.

Speculative decoding: MTP and DFlash

Atlas ships two speculative paths, mutually exclusive per serve:

• MTP draft heads (--speculative) — the checkpoint’s own multi-token-prediction head proposes K tokens per step, verified with WY-chunkwise GDN kernels. K=2 is the measured-fastest width on Qwen3.5-35B-A3B (the 131 tok/s row above).
• DFlash block diffusion (--dflash) — pairs the target with a small drafter (e.g. z-lab/Qwen3.6-35B-A3B-DFlash) that emits γ tokens per step via bidirectional in-block attention conditioned on captured target hidden states (Z Lab, arXiv:2602.06036). Atlas serves the measured record shape by default: γ resolves to the drafter’s trained block size + 2.

Honest status of DFlash on GB10: with the DFlash2 drafter for Qwen3.8-27B at default flags, Atlas reaches 66.6 tok/s at C=1 — a single-stream number, and only that. At higher concurrency DFlash2 is currently a net loss on this hardware (measured −7.1% at C=8 and −29.1% at C=16 versus the same engine without the drafter). That gap is a known, tracked open item, and the concurrency-sweep-dflash2 gate runs the full ladder with the drafter armed on every relevant PR so it cannot regress silently. If you serve concurrent agent traffic today, MTP or the plain engine is the right choice.

Kernel-level receipts

The kernel-by-kernel comparison against PyTorch eager lives in the benchmarks chapter along with the methodology footnotes — read them; they matter. That table is 32 benchmark rows over ~11 kernel families (attention, GEMM, W4A16, MoE, conv1d, GDR, RMSNorm, SiLU×Mul, RoPE), all wins; it is not a sweep of the whole registry. The registry itself is much larger — kernels/gb10/common/ alone holds ~170 .cu files, before the per-model shadow directories.

🗜️ KV Cache Quantization

Atlas stores attention key/value state in a quantized format selected via --kv-cache-dtype. Lower bit-widths fit more tokens in GPU memory at the cost of precision; the Turbo family adds Walsh-Hadamard rotation and Lloyd-Max optimal codebooks to recover accuracy at the same bit rate. Mix dtypes per layer with --kv-high-precision-layers to keep boundary layers at BF16 while compressing the middle.

The table above is the symmetric set — the same format for K and V. KvCacheDtype (crates/spark-runtime/src/kv_cache.rs) accepts 16 values in total: the six above plus turbo2 (2-bit) and nine TurboQuant+ asymmetric K/V pairings (turbo4k_turbo3v, turbo4k_turbo8v, turbo3k_turbo8v, bf16k_turbo4v, bf16k_turbo3v, bf16k_turbo2v, fp8k_turbo4v, fp8k_turbo3v, fp8k_turbo2v) that store K at higher precision than V, since K dominates attention-score fidelity. Those are documented in docs/turboquant-plus.md; the parser is the authority on the accepted spelling.

🏛️ Architecture

The diagram below shows how a single HTTP request flows from the API surface down to hardware-specific CUDA kernel execution. Dashed borders mark the plug-and-play abstraction boundaries — the traits and registries where a new hardware target, model family, communication backend, or storage backend plugs in without touching the layers above or below it.

Reading the diagram

Solid boxes are concrete implementations. Dashed borders with 🔌 are the trait-based abstraction boundaries — each is a Rust trait (or a filesystem convention for kernels) where a new integration plugs in:

Data flow summary

1. HTTP → spark-server receives OpenAI/Anthropic requests, tokenizes, and enqueues
2. Scheduler → batches sequences, orchestrates prefill/decode/speculative-verify steps
3. Model → generic loop: embed → [layer₀ … layerₙ] → norm → lm_head
4. Layers → each layer dispatches through GpuBackend to launch kernels from AvarokRegistry
5. Kernels → pre-compiled PTX selected by (hardware × model × quant) target at build time
6. EP → CommBackend handles cross-GPU all-reduce after MoE expert computation
7. Storage → StorageBackend spills/restores KV blocks to NVMe for long-context sequences

🧭 Why Atlas Exists

Atlas began as a response to a widely felt problem with Python inference stacks: a shifting ecosystem of dependencies, patches, and cross-dependencies where the workaround that ran your model yesterday needs a nightly branch and a new workaround today. That is how you build a proof of concept, not a software ecosystem. We are grateful to the data scientists who proved what LLMs can do; Atlas is the software engineers taking the torch and building the version designed to withstand the test of time. The full argument is in Seven Tenets Powering Atlas Inference on the blog; the short version:

Similar to how llama.cpp was built to prove you don’t need five- or six-figure GPUs to run LLMs, Atlas exists to keep forcing the narrative that as hardware advances, inference should not cost premium cloud-API prices. Maximizing speed for each hardware/model combination is what makes meaningfully powerful LLMs truly useful on hardware you own.

AI-authored PRs are the default, and the target

This codebase was built with enough guardrails, structure, and abstraction to let an AI absorb the monorepo and contribute meaningfully — which means that instead of waiting weeks for model support, you can fork this repo, point your agent at it, and more likely than not have a working model within hours. If you write code by hand, we ask you to say which parts and why the human beat the AI — not to discourage you, but because every such case marks a gap in the tooling we would rather close than live with.

The contribution loop has exactly two exits — merge, or back to editing:

The per-state commands, exit conditions and the invariants an agent must not violate are in CONTRIBUTING.md — in a table, because an agent should not have to infer the contract from prose.

🔌 Adding a New Hardware Target

The full recipe is in docs/HARDWARE.md. The short version: implement two traits (ComputeTarget for the build-time compiler, GpuBackend for the runtime), drop kernel sources into kernels/<your-hw>/, add one match arm in the registry. There is a MockGpuBackend in spark-runtime that lets you write and test the entire scaffold without owning the hardware — every layer above the GPU trait is hardware-agnostic, so unit tests can run on a laptop. We bolted the project from “single CUDA target” to “trait-pluggable across vendors” specifically so that the next ports stop being our problem and start being yours — and the Strix Halo port under kernels/strix/ is the worked example.

🧬 Adding a New Model

Same story, smaller surface. Implement ModelWeightLoader (one struct; the existing Qwen3AttentionLayer/MoeLayer/Qwen3SsmLayer/NemotronMamba2Layer primitives cover most architectures), add one line to the factory dispatch, optionally drop a MODEL.toml for sampling defaults and behavior knobs. Kernels are reused; the scheduler is untouched; the server is oblivious. The step-by-step cookbook is in docs/HARDWARE.md. Once your loader produces coherent output on the integration coherence prompt, you are done — file the PR.

🔬 Kernel Debugging

Atlas exposes a focused set of environment-gated diagnostic dumps for tracking down quality regressions — magnitude drift, expert-routing skew, MoE under-counting, and the rest of the bug class where the kernels run cleanly but the output slowly degrades. The dumps are zero-overhead when their env var is unset (single var() lookup per call, no GPU sync, no copy) so leaving the production binary instrumented is safe.

For the full diagnostic playbook — including the cheapest-signal-first elimination ladder, how to build a byte-exact HF CPU oracle, the per-layer divergence comparator, and the methodological reversals that cost us hours — see DEBUGGING_METHODOLOGY.md. What follows is the env-var reference.

MoE-path dumps — AVAROK_DUMP_EXPERT_IDS=1

Set -e AVAROK_DUMP_EXPERT_IDS=1 on the container. The markers themselves live in one place — crates/spark-model/src/layers/moe/dump.rs — and every MoE path calls into it (forward_prefill.rs, forward_prefill_fp8.rs, forward_prefill_bf16.rs, forward_prefill_routed.rs, forward_batched.rs). They emit the following per-fire log lines, scoped to the last token of the chunk so the values are directly comparable to a single-pass reference forward at the same position:

SSM-path dumps

The SSM (GDN / Mamba-2) prefill in qwen3_ssm/trait_prefill.rs adds three pre-norm hooks under the same env var:

Together, those plus the MoE dumps above give a complete trace of the residual stream at every layer boundary for any token in any chunk.

Path-toggle env vars

For bisecting which code path is at fault, one override toggle lets you swap the routed-expert dispatch at runtime without rebuilding:

There is no longer an FP8 grouped-GEMM v1/v2 selector: moe_fp8_grouped_gemm is a single grid-compaction kernel (kernels/gb10/common/moe_fp8_grouped_gemm.cu), and the AVAROK_FP8_MOE_COALESCED gate that once chose between them has no read site in the tree.

How we use these in practice — 3-step workflow

The order matters; this is the same workflow that found and fixed three compounding MoE bugs (commits 6a5fd3d, 34626d3, adf39ce, ffdb41d) on the Qwen3.6-A3B long-context investigation:

1. Build an HF reference oracle. A single-precision forward pass through HF Transformers on the same token IDs (read them back from Atlas’s /tokenize — do not re-render the chat template), with output_hidden_states=True and per-layer hooks on mlp.gate, mlp.shared_expert, and mlp.shared_expert_gate. Record \|x\| + first5 per layer for the last token.
2. Spin up Atlas with -e AVAROK_DUMP_EXPERT_IDS=1. Fire the same prompt. The MoE markers above give you per-layer Atlas values comparable to the oracle.
3. Per-layer comparator. A short script (the comparator pattern is captured in DEBUGGING_METHODOLOGY.md §4) prints ratio = |Atlas| / |HF| and overlap = |top-K_Avarok ∩ top-K_HF| per layer. The first layer where the ratio falls outside [0.95, 1.05] or overlap drops below 6/8 is your first-divergent layer — start drilling there.

For the 2026-05-20 MoE bug hunt this localized the issue from “16K context produces gibberish” to “L0 MoE output magnitude 3.4× too large because of three compounding bugs: v1 grouped-GEMM, missing zero-init, broken max_m_tiles heuristic” within a few iterations. After all three fixes, all 40 layers landed in [0.977, 1.021] of HF baseline — at the FP8 quantization noise floor.

🔐 How a Change Lands

Every claim in this README is a measurement, and measurements are only worth the process that produced them. A change reaches main through three stages, and nothing about that is manual goodwill — it is enforced.

Stage 1 — Verification runs on every push: formatting, clippy, typos, licence headers, kernel structure, tests, merge ancestry. Certification and the nine release-matrix build legs are held back, so an early draft does not burn an hour of runners.

Stage 2 — Certification opens on /stamp from anyone with write access or the PR’s own author — nobody is better placed to say their own branch has stopped churning, and the author is precisely who the hold was protecting from burnt runners. It needs two things. An engineer’s seal — a codeowner comments /seal, and the sealers’ owned paths must cover the whole diff. And benchmark records — the campaign runs on a real GPU box and the records are committed; CI only checks that they cover what changed. Records are Ed25519-signed over their own bytes and the commit they name, and CI requires every record a PR adds to share one commit and one signer — so an altered record, one re-pointed at a different commit, or results spliced from two campaigns are all detectable, and every record is attributable to a key in .github/record-signers/.

That is an integrity check, not proof the numbers were measured. The key is generated on the operator’s own box and signing happens there, after the record is written, so a determined insider can sign whatever they like; CI has no GPU and verifies signatures and internal consistency rather than witnessing execution. What this makes nearly impossible is the accident — shipping the wrong records — and what it makes attributable is everything else.

Stage 3 — Ready to merge, then the queue, which re-runs the whole pipeline against its own merge commit.

The bot takes five comment commands, and they are the whole interface:

/expedite exists because a release should not be hostage to a GPU box being busy. It is deliberately loud: it mints its own check run, posts a comment naming who used it and why, and the reason is required rather than optional.

The asymmetry in the table is the part worth reading twice. A seal survives main moving — the sealer vouched for this diff, and the queue re-runs everything against the composed tree. Records never do. Two campaigns measured apart do not compose: the interaction between them was never measured, and calling it measured is the exact class of silent regression the gate exists to prevent. Freeze the branch, run the campaign, queue alone.

🤝 Community and Contributing

The action is in Discord — we are in there every day, shipping fixes, taking model requests, and tuning kernels in the open. Your machine is the test fleet and your voice sets the roadmap.

• Run the serve matrix on your own hardware and report what you see — regressions and wins both get featured. Start with the GB10 Deployment Guide.
• Add or tune a recipe in atlas-recipes — recipes are the model SSOT.
• Write kernels in Rust and CUDA — hand-tuned attention, MoE, GDN, Mamba-2 for Blackwell. Register-level work, no generic fallbacks.
• Docs, triage, ideas — improve the guides, triage issues, or open a thread in Discussions.
• Follow along on the blog and on X @AtlasInferenceX.

Contributor workflow, code standards, and the PR gate contract are in CONTRIBUTING.md. Contributions ship in the Community Edition under AGPLv3, and the CLA permits re-licensing for the Enterprise Edition. Please also see the Code of Conduct and the security policy.

📚 Citations

We did not invent the kernels we ship. We picked the right ideas from the right papers, fused them together, and tuned them for one chip until they pinned the bandwidth ceiling. Atlas owes a direct intellectual debt to:

• FlashAttention-2 — Tri Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024. arXiv:2307.08691 — tiled online softmax, Q/K/V SMEM staging, causal masking. Foundation of our prefill kernel.
• FlashAttention-4 — Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao. FlashAttention-4: Taming the Hardware. 2025. arXiv:2603.05451 — conditional softmax rescaling and software polynomial sw_exp (3 FMA + ldexpf instead of going through the SFU). Both shipped in our GQA-fused paged Flash Attention.
• FlashInfer — Ye, Chen, Lai, Zhao, Zheng, Shao, Hou, Jin, Zuo, Yin, Chen, Ceze. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. MLSys 2025 (Best Paper). arXiv:2501.01005 — block-sparse paged KV cache, page index prefetch to SMEM, the gather-SMEM-MMA pattern for scattered pages. Informed our paged attention design.
• SageAttention 3 — Zhang, Huang, Zhang, Wei, Zhu, Chen. SageAttention3: Microscaling FP4 Attention on Blackwell GPUs. NeurIPS 2025 Spotlight. arXiv:2505.11594 — FP4 attention with FP8 per-block microscales. On the SM121 roadmap once silicon-level FP4 MMA arrives upstream.
• LeanAttention — Roy, Vassilieva, Willke, Mendis. LeanAttention: Hardware-Aware Scalable Attention for LLM Inference. 2024. arXiv:2405.10480 — stream-K tile scheduling for near-100% SM occupancy in split-K decode attention. Planned next.
• DFlash — Z Lab. Block-diffusion speculative decoding. arXiv:2602.06036 — a small drafter emits γ tokens per step via bidirectional in-block attention conditioned on captured target hidden states. The basis of Atlas’s --dflash speculative path.
• TurboQuant — Zandieh, Daliri, Hadian, Mirrokni. TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate. arXiv preprint, April 2025. arXiv:2504.19874 — Randomized Hadamard Transform + Lloyd-Max codebook for KV cache compression. The implementation in our kernels/gb10/common/wht_bf16.cu + reshape_and_cache_turbo.cu follows the TurboQuant+ extensions (matched-norm L2 correction, sparse V dequant, asymmetric K/V, InnerQ per-channel equalisation) collected at TheTom/turboquant_plus (research umbrella) with the llama.cpp engine reference at TheTom/llama-cpp-turboquant; per-feature reproduction and prior-art chain in docs/turboquant-plus.md.

The full acknowledgment list is in CITATIONS.md. If you wrote one of these papers and you spot a misattribution or a wrong technique credit on our side, open an issue. We would rather be corrected than wrong.

⚖️ License and Enterprise Edition

Atlas operates under a dual-license model. Both are real, both are intentional, and neither is a teaser for the other.

1. Community Edition — AGPLv3. Free, open, copyleft. Use it for yourself to run inference on your own hardware, research, hobby projects, side-projects, and/or hosted demos, as examples. If you want to make money from Atlas, purchase a commercial license.
2. Enterprise Edition — commercial license. If you need to ship Atlas inside a closed-source product, run it as a SaaS backend without inheriting the AGPLv3 source-disclosure obligation, or simply want a support relationship with the people who wrote the kernels, contact us. Enterprise customers also receive prioritized model and hardware ports.

This split exists for a single reason: the commercial license keeps us building Atlas full-time, and the AGPL community license keeps the project honest. Contributions are covered by the CLA, which permits Enterprise re-licensing while you retain ownership of your contribution. What is in this repository is what we run.

atlascybernetics.ai · docs.atlascybernetics.ai · blog.atlascybernetics.ai

_{The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See mlcommons.org for more information.}