disaggregated-serving 开源项目推荐（17 个）

ai-dynamo /dynamo

数据中心级分布式推理服务框架

DeepLink-org /DLSlime

Composable and Embeddable Communication Runtime for Distributed AI Services

athrael-soju /Narwhal

Adaptive Disaggregated Inference on a Role-free Fleet!

awneesht /KVShuttle

Benchmark & decision framework for KV cache transfer compression in disaggregated LLM serving — 14+ compressors, 6 models, GPU-calibrated timing, decision flowchart

developertogo /velo-core

A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.

JohnScheuer /hybrid-inference-runtime

Disaggregated Prefill-Decode serving engine connecting NVIDIA Blackwell (SM120) and Huawei Ascend 910B2 (CANN) across physical nodes.

developertogo /velo-sentinel

Production-grade Java 25 Virtual Thread inference gateway bridging NVIDIA Triton → Dynamo with Earliest Deadline First (EDF) priority queuing, adaptive batching, and async shadow validation.

richajoy /moe-disaggregation-benchmark

Does prefill/decode disaggregation pay for an MoE model? SLO-first goodput benchmark of Qwen3-30B-A3B on 4x H100 and 8x A100 with vLLM, llm-d and NVIDIA Dynamo, with per-request data.

sunghyunjun /pd-disaggregation-over-tcp

Chunked prefill vs. prefill-decode disaggregation over TCP on single-GPU cloud instances: measurements, calibrated model, and per-request data (vLLM 0.30.0, NIXL/UCX, Gemma 4). Technical report doi:10.5281/zenodo.23085600

Devil-SX /splitwise-modeling

Reproducible capacity and throughput modeling for co-located vs disaggregated LLM Prefill/Decode serving

Shakhtar-Sankur /relay

该项目在 GitHub 上没有填写描述

MicheleCampi /vllm-disagg-profiling

A/B profiling: aggregated vs disaggregated prefill/decode LLM serving on equal hardware, measured per-device with inferscope

varad-more /kv-transfer-tax

Measuring the KV-transfer tax of disaggregated prefill/decode LLM serving (vLLM + NIXL, 4x A10G): on PCIe-only hardware, disagg loses to plain data-parallel replication — measured, committed, reproducible.

JohnScheuer /prefix-sharing-disaggregated-bench

Prefix KV sharing in disaggregated LLM serving: transfer vs recompute vs hotset/LRU replication under decode memory pressure.

LilySu /llm-inference-gateway-aws

A fault-tolerant LLM routing system that decouples inference from AWS Bedrock by routing prefill and decode tasks through SQS and ensuring zero-downtime scaling with a graceful drain sidecar.

bellshaped-internalsecretion2730 /DYB-Pro

Automate protein design cycles from research goals to ranked wet-lab shortlists with git-like version history.

BrendanJamesLynskey /Disaggregated_Inference_Sim

SimPy simulator of prefill/decode-disaggregated LLM inference: latency, hot-spots, utilisation, power & energy, exact acceleration, 32 tests — companion to the LLM Inference Simulators series