llms-benchmarking 开源项目推荐（112 个）

steel-dev /awesome-web-agents

🔥用于构建AI Web代理的工具、框架和资源列表

karthikv792 /LLMs-Planning

An extensible benchmark for evaluating large language models on planning

JonathanChavezTamales /llm-leaderboard

A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)

bboylyg /BackdoorLLM

[NeurIPS 2025] BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models

lechmazur /nyt-connections

Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words

ChemFoundationModels /ChemLLMBench

Official Code for What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks (In NeurIPS 2023)

lamalab-org /chembench

How good are LLMs at chemistry?

Portkey-AI /models

This repository contains comprehensive pricing and configuration data for LLMs. It powers cost attribution for 200+ enterprises running 400B+ tokens through Portkey AI Gateway every day.

lerogo /MMGenBench

Official repository of MMGenBench

parea-ai /parea-sdk-py

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

SuperBruceJia /Awesome-Mixture-of-Experts

Awesome Mixture of Experts (MoE): A Curated List of Mixture of Experts (MoE) and Mixture of Multimodal Experts (MoME)

FSoft-AI4Code /XMainframe

大型机现代化的语言模型

lechmazur /generalization

Thematic Generalization Benchmark: measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.

eth-sri /constrained-diffusion

Constrained Decoding of Diffusion LLMs with Context-Free Grammars.

mbzuai-oryx /Agent-X

ICLR 2026: Agent-X Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks

RaptorMai /MLLM-CompBench

[NeurIPS'25] MLLM-CompBench evaluates the comparative reasoning of MLLMs with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes

multinear /multinear

Develop reliable AI apps

RUC-GSAI /YuLan-SwarmIntell

🐝 SwarmBench: Benchmarking LLMs' Swarm Intelligence

amazon-science /llm-code-preference

Training and Benchmarking LLMs for Code Preference.

declare-lab /resta

Restore safety in fine-tuned language models through task arithmetic