multimodal-llm 开源项目推荐（198 个）

FireRedTeam /FireRedASR

开源工业级 ASR 模型，支持普通话、中文方言和英语，在公开普通话 ASR 基准上取得新的 SOTA，同时具备出色的歌词识别能力。

Tencent /WeMM-Embedding

WeMM-Embedding is a family of universal multimodal embedding models by the WeChat Vision Team at Tencent, supporting multimodal understanding and retrieval.

gokayfem /awesome-vlm-architectures

Curated visual catalog of 200+ vision-language model (VLM/MLLM) architectures: papers, diagrams, training recipes, datasets, and a release timeline for multimodal AI agents.

UCSB-AI /MiniGPT-5

Official implementation of paper "MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens"

FireRedTeam /FireRedASR2S

一套SOTA工业级一体化ASR系统，配备ASR、VAD、LID和Punc模块。FireRedASR2 支持中文（普通话，20+ 方言/口音）、英语、语码切换，以及语音和歌唱 ASR。FireRedVAD支持100+语言的语音/唱歌/音乐。FireRedLID 支持 100+ 语言和 20+ zh 方言。FireRedPunc支持zh和en。

v-modal /vmodal_sdk_robotics

Robotics vision SDK

AI45Lab /OpenRT

Open-source red teaming framework for MLLMs with 42+ attack methods

xmed-lab /TAM

[ICCV25 Oral] Token Activation Map to Visually Explain Multimodal LLMs

alipay /Ant-Multi-Modal-Framework

蚂蚁集团多模态认知团队的研究代码

yuxin-jiang /Anomagic

[AAAI 2026] The Official Implementation for "Anomagic: Crossmodal Prompt-driven Zero-shot Anomaly Generation"

HVision-NKU /OraRL

🎬 OraRL — Annotations as Rollouts for efficient, scalable reinforcement learning of unified video MLLMs.

InfiXAI /InfiGUI-G1

[AAAI 2026 Oral] Official repository for InfiGUI-G1. We introduce Adaptive Exploration Policy Optimization (AEPO) to overcome semantic alignment bottlenecks in GUI agents through efficient, guided exploration.

ProgressLM /ProgressLM

[ACL 2026 Oral] How far can VLMs go in understanding long-horizon action progress from just a single observation? A promising early exploration

hanxunyu /DepthVLM

🔥 Official code repository for "Unlocking Dense Metric Depth Estimation in VLMs"

AarambhDevHub /aarambh-studio

🦀 Decoder-only LLM built from scratch in pure Rust using Candle — no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE, native video/document understanding, long-horizon tool agents, quantization-aware training. Scales: Tiny (25M) to Large (1.3B).

Zhoues /MineDreamer

[IROS'25 Oral & NeurIPSw'24] Official implementation of "MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control "

UCSC-VLAA /vllm-safety-benchmark

[ECCV 2024] Official PyTorch Implementation of "How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs"

NVIDIA /sports-intelligence-playbooks

This repository provides a starter kit for sports intelligence built on NVIDIA AI stack: playbooks, training recipes, and inference for Multimodal Language Models.

HarryHsing /OmniAgent

OmniAgent (ICML 2026): the first native omni-modal agent for active video perception — a 7B agent that beats Qwen2.5-VL-72B with 73% fewer frames on LVBench.

xiaofengShi /MechVQA

ICML 2026 benchmark and MechVL models for multimodal understanding of mechanical engineering drawings.