inference-platform 开源项目推荐（10 个）

bentoml /BentoML

部署 AI 应用和模型的最简便方式——构建模型推理 API、任务队列、LLM 应用、多模型流水线等！

InftyAI /llmaz

☸️ 一个简单、先进的大型语言模型推理平台，在Kubernetes上。🌟 星星支持我们的工作！

Lattica-ai /lattica-studio

A tool for developers and researchers to design FHE pipelines and execute them on cloud-based GPUs

MrScripty /Pantograph

Local AI workflow orchestration and runtime management framework

lee-cheng-han /gpu-aware-llm-serving-platform

GPU-aware LLM serving platform with multi-worker routing, dynamic batching, model placement, streaming, admission control, and reproducible performance benchmarks.

manishklach /InferenceFlow

Prepaid asynchronous LLM inference platform: FastAPI, PostgreSQL-backed queues, deadline-aware scheduling, modular GPU workers, vLLM integration, token metering, ledger billing, Stripe test payments, Next.js dashboard, Python SDK, and Docker. Includes a mock mode and recorded walkthrough; real GPU validation pending.

TheStableFoundation /stbl-protocol

该项目在 GitHub 上没有填写描述

gauravkukal /sensei-agentic-inference-orchestrator

A conceptual framework for a high-scale Agentic AI orchestrator, inspired by enterprise-grade inference platforms.

Sourajit-M /local-ai-inference-platform

Local AI inference platform built with FastAPI, React, Ollama, and SQLite. Features multi-model benchmarking, structured generation, experiment tracking, JWT authentication, and performance analytics for small language models running entirely offline.

CeciliaGit /ai-inference-platform-lab

AI inference platform architecture lab demonstrating admission control, fairness scheduling, bounded queues, and graceful degradation under burst traffic.