ai-evals 开源项目推荐（127 个）

microsoft /PyRIT

微软开源 AI 红队测试工具包。通过自动扫描、自适应多轮攻击和 Web UI 测试生成式 AI 系统。

future-agi /future-agi

用于评估、观察和改进LLM和AI代理应用程序的开源端到端平台。追踪· Evals ·模拟·数据集·门户·护栏。可自托管。阿帕奇2.0。

modiqo /skillspec

SkillSpec makes agent skills followable, testable, and provable with Doctor risk reports, guided imports, structured contracts, and alignment proof.

coval-ai /benchmarks

Reproducible voice-AI benchmarking — TTS / STT / S2S latency and accuracy.

Calibre-Labs /reforge-ai-evals

Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.

aisa-group /InferenceBench

Benchmarking Open-Ended Inference Optimization by AI Agents

Stunspot /stunspots-guide-to-ai-systems

Operational doctrine for practical AI systems design.

danielrosehill /Awesome-AI-Evaluations-Tools

Collection of frameworks and tools for AI evalations, including tool-use, agentic AI, MCP, and multimodal

parthmax2 /ghostrun

pytest for AI Apps — Fast, free, offline testing and self-improving prompts for Python developers.

ARTPARK-SAHAI-ORG /calibrate

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

solana8800 /langeval

Evaluation Infrastructure for AI Agents

jimy-r /agent-workspace-architecture

Harness engineering for Claude Code, run daily: hooks, skills, typed memory, roles, a delegation queue and self-audits that let you hand a coding agent work and trust it while you're away. 18 load-bearing patterns, an interactive tour, a learn track, teardowns of real systems, a CI linter and fork-ready samples.

productfoundry101 /ai-evals-bootcamp

Learn to evaluate AI products for production — 21 hands-on lessons on evals, metrics, fairness, agents, red teaming, and release decisions for working PMs.

yiouli /pixie-qa

Agent skill for AI agent development

RyanAlberts /pmstack

Gstack for PMs. Eval Studio that you can plug-n-play anywhere.

evalibrate /openjury

An sdk and framework for evaluating and comparing multiple model outputs using configurable LLM-based jurors

selectess /fde-consultants-protocoles

FDE Consultants Protocoles — open-source Forward Deployed Engineer + DeepSCR protocol: turn any coding agent (Claude Code, Codex, Cursor) into a certifying engineer with verifiable AI Assurance Scores, MCP tools, and a public trust registry. Apache-2.0.

mohsinsheikhani /property-maintenance-agent

Eval-first AI agent that triages property maintenance emails. The real work is the eval system around it: trace-driven error analysis, code graders and validated LLM-as-judge (TPR/TNR), component and end-to-end evals, a failure taxonomy, and a CI regression gate. LangGraph, FastAPI, Langfuse.

acblabs /multi_llm

Governed multi-LLM Responsible AI control plane for prior-authorization decision support, with PHI-safe audit trails, HITL review, evidence packets, and deterministic governance evals.

vibheksoni /jailbench

Benchmark LLM jailbreak resilience across providers with standardized tests, adversarial mode, rich analytics, and a clean Web UI.