ai-evaluation 开源项目推荐（1,636 个）

ifixai-ai /iFixAi

AI 智能体的独立审计。可由人类或智能体自身运行，以回答 AI 智能体经济中最关键的问题：智能体是否在做它应该做的事？使用 iFixAi，你可以在不到 120 秒内得到答案。

microsoft /PyRIT

微软开源 AI 红队测试工具包。通过自动扫描、自适应多轮攻击和 Web UI 测试生成式 AI 系统。

cvs-health /uqlm

UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection

Simreal-AI /Simreal-MLBench

[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

lechmazur /confabulations

Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

guestrin-lab /deepscholar

build and benchmark deep research

rungalileo /agent-leaderboard

在智能体任务上对 LLM 进行排名

jsdhwfmax /EvalForge

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

EastForce /worker-ai-spark

开源劳动者AI研究与评测项目：建设劳动知识、公开评测基准与共同治理机制｜Open worker-centered AI research and evaluation project.

mahmoudrabie /agentic-ai

Agentic AI research papers, benchmarks, frameworks, and tools curated across 24 domains.

suncityldp /zx-bench

A reproducible local LLM benchmarking platform with versioned datasets, deterministic scoring, and auditable reports.

huggingface /cadgenbench

A benchmark for AI-driven CAD generation and editing

METR /vivaria

Vivaria is METR's tool for running evaluations and conducting agent elicitation research.

evaleval /every_eval_ever

Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderboard scrapes and research papers to local evaluation runs — so that results from different frameworks can be compared, reproduced, and reused.

ianarawjo /evalstats

Statistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.

taoAIGC /AICompare

one click to open multi AI sites ｜ 一键打开多个 AI 站点，查看 AI 结果

Vvkmnn /awesome-ai-eval

Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.

Simreal-AI /hard-puzzle-benchmark

[Public preview] Hard human-written reasoning puzzles: 749 answered items in 15 topics (707 scored core), with protocol and scorer.

Simreal-AI /Test

[Public preview] The first public quant trading benchmark scored against what professional traders actually made on the same asset

agentvitals /checkup

AgentVitals Checkup (/checkup) — an AI agent skill that gives your agent a professional health checkup: dual-axis Stability + Welfare scoring, a personality-style title, and a public cross-platform leaderboard. One-line install for Claude Code / OpenClaw / Codex / Coze. Bilingual EN/ZH.