llm-validation 开源项目推荐（27 个）

security-decision-science /security-decision-labs

FAIR网络风险量化工具包、基于代理的控制仿真（FAIR-CAM）、威胁事件频率估计器（PyPI）、大型语言模型分类验证器（PyPI）、带有IRIS基准的蒙特卡洛风险引擎。

Vedantgitbot /booth

A lightweight, zero-dependency Python checkpoint layer for structured LLM output reliability.

Oncoshot /llm-validation-framework

A comprehensive Python framework for evaluating LLM-extracted structured data against ground truth labels. Supports binary classification, scalar values, and list fields with detailed performance metrics, confidence-based evaluation, and statistical uncertainty quantification via non-parametric bootstrap confidence intervals.

sazed5055 /llmtest

pytest for LLM apps - Test for grounding failures, prompt injection, safety violations, and regressions

cengicadis /blankvisuals-validator

Evidence-backed validation framework for AI agents, Cursor skills, MCP orchestrations, and agentic repos. Structured rule catalog, verified evidence standard, and certification output. Free, open-source, MIT.

mattijsmoens /logicshield

Deterministic validation firewall that verifies AI-generated proposals against ground-truth state using immutable rules. Zero dependencies.

ajdedeaux /ai-eval-framework

Systematic AI evaluation framework that transforms subjective assessment into objective measurement. Reduce research time by 85% while maintaining 95%+ accuracy through multi-LLM validation.

ekberglpnar /formwork

Runtime semantic constraints and targeted repair for structured LLM output — declare which fields your code owns and which the model may invent.

chigwell /llmtestr

A new package that helps developers integration-test AI and LLM applications by validating structured outputs. It takes a user's test scenario or prompt as input, sends it to an LLM, and uses pattern

rudrakshkarpe /dual-llm-bench

Setting benchmarking standards for the LLM security!

Ali-datasmith /Marketing-ROI-Tracker

Enterprise B2B multi-touch attribution and budget optimization engine built with Polars, DuckDB, Pydantic v2, Gemini structured output, and Streamlit.

VidhyaSangeetha /ai-response-quality-validator

Production-grade AI quality gates for LLM applications. Automating hallucination detection, semantic accuracy scoring, and adversarial safety testing for RAG & Agentic AI systems.

maxwellwilber-cpu /fas-case-study

27-node AI financial analysis pipeline — case study on validation architecture, fail-closed design, and real-world results

flam7791 /oecd-data-pipeline

Public economic indicators (SDMX API) to cleaned data to Copilot-written country notes, each checked against strict rules before it is accepted. Independent project, not affiliated with the OECD.

DimaDoesCode /LLM-Evidence-Update-Validation

Can an LLM revise a decision when new evidence warrants a reversal - while remaining stable when the evidence does not warrant one?

workfreelance28-afk /pharmacheck-ai-portfolio-case

Sanitized portfolio case: safe AI integration for drug interaction analysis with LLM validation, Telegram flow and serverless architecture.

anant-pw /n8n-local-llm-qa-agent

Local-LLM agentic pipeline (n8n + Ollama) that turns a raw requirement into validated QA test cases — two independent validation layers (semantic + structural), zero cloud dependency, runs on a 16GB CPU-only laptop. Includes a documented hallucination failure case and honest known-limitations section.

MantisAI /GSMA-dataset-creation

GSMA telecom documentation dataset creation pipeline with hard negative generation for embedding training. Features concurrent LLM validation, semantic similarity ranking, and DVC-based reproducible data processing.

tk-yasuno /patchtst_pareto_dgm

The apps is a Darwin-Gödel Machine self-improvement framework to automate hyperparameter tuning for a PatchTST-based time-series anomaly detector. This optimizes four Focal Loss and class-weight parameters across 15 objectives (AUC, Precision, Recall, F1, FPR × 3 horizons) using NSGA-II Pareto multi-objective optimization with LLM-based validation.

Fauxetine /t1-t2-protocol

MCP server for heterogeneous AI validation — T1 structured reasoning, T2 cross-model evaluation, and deterministic checksum. Python stdlib-only.