RAG Financial Multimodal — Enterprise v2.0
Production-grade multimodal RAG for financial document intelligence.
Chart understanding · hybrid retrieval · numeric guardrails · multi-tenancy · full observability.
About the history: this codebase was rebuilt from scratch in 2026. The commits from
February 2024 at the start of the history are an earlier, unrelated upload: a notebook,
two pickled data files and an image. Nothing in the rebuild used them, so they were removed
from the tree in October 2026. They remain in the history (commits 781d919–1d08a81).
System Architecture
Ingestion (top) and query (bottom) pipelines. Every component is pluggable — switch provider by changing one config value.
Why This System
Financial documents are mixed-media: narrative text, tables, charts, footnotes, cross-references. Standard RAG pipelines fail on charts and hallucinate numbers.
| Problem |
Solution |
| Charts contain the most important data but RAG ignores them |
GPT-4o / Gemini / Qwen2-VL vision extraction — every chart yields exact axis values |
Exact figures like $23.35B or TSLA miss semantic search |
Hybrid RRF: dense embeddings + BM25 keyword fused with Reciprocal Rank Fusion |
| LLMs fabricate financial numbers |
Numeric grounding guardrail — every stated number verified against source context |
| PII in analyst queries leaks to APIs |
Presidio + CUSIP/ISIN/account number redaction before any external call |
| One broken vendor = full outage |
Fallback chains — primary → secondary → local for every model-facing layer |
Retrieval Quality
Measured offline on the 16 real filings behind the golden dataset (119 questions), using this
repo’s BM25, local bge-small embeddings and RRF fusion with default settings and no reranker.
Evidence hit@5 means a top-5 chunk contains the exact supporting text for the answer.
| Strategy (4,000-char chunks, default) |
Evidence hit@5 |
Doc hit@5 |
| BM25 only |
0.71 |
0.87 |
| Dense only |
0.61 |
0.99 |
| Hybrid RRF |
0.71 |
0.93 |
| Hybrid RRF + knowledge graph (optional) |
0.72 |
0.90 |
With 1,000-character chunks, hybrid reaches 0.53 versus 0.42 (BM25) and 0.46 (dense), so fusion
helps there. At the default chunk size it only ties BM25. About 3 in 10 questions get no
answer-bearing chunk in the top 5. Full results, method and caveats are in
BENCHMARK_RESULTS.md. Earlier versions of this README claimed 89%
Recall@5. That figure could not be reproduced and has been withdrawn; the original charts are
kept, with a provenance note, in BENCHMARK_RESULTS.md section 4.
Quickstart
git clone https://github.com/Mattral/RAG-Multimodal-Financial-Doc-Analysis-and-Recall
cd RAG-Multimodal-Financial-Doc-Analysis-and-Recall
cp .env.example .env # set OPENAI_API_KEY or GOOGLE_API_KEY
docker compose up -d
curl -X POST http://localhost:8000/api/v1/ingest \
-F "file=@tesla_10k.pdf" -F "tenant_id=demo"
curl -X POST http://localhost:8000/api/v1/query \
-H "Content-Type: application/json" \
-d '{"query": "What was gross margin in Q3 2023?", "tenant_id": "demo"}'
# Same query, answer streamed token by token (server-sent events)
curl -N -X POST http://localhost:8000/api/v1/query/stream \
-H "Content-Type: application/json" \
-d '{"query": "What was gross margin in Q3 2023?", "tenant_id": "demo"}'
Streamed tokens are provisional: guardrails run on the full answer and are reported in the
final done event (REST API docs).
Or use the CLI:
pip install -e ".[all]"
rag-financial ingest tesla_10k.pdf --tenant demo
rag-financial query "What was Q3 revenue?" --tenant demo --show-sources
Multi-Provider Support
Every model-facing layer (text generation, vision, embeddings, vector store) is independently pluggable. Switch via a single .env line — zero code changes.
Fully open-source / zero-API-cost configuration
LLM_CONFIG__PROVIDER=local_vllm
LLM_CONFIG__MODEL=meta-llama/Llama-3.1-8B-Instruct
LOCAL_VLLM_GENERATOR_BASE_URL=http://localhost:8090/v1
VISION_CONFIG__PROVIDER=local_vllm
VISION_CONFIG__MODEL=Qwen/Qwen2-VL-7B-Instruct
VECTOR_STORE_CONFIG__EMBEDDING_PROVIDER=local
VECTOR_STORE_CONFIG__EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8090 --host 0.0.0.0
vllm serve Qwen/Qwen2-VL-7B-Instruct --port 8080 --host 0.0.0.0
Evaluation Quality
-
Golden dataset: 119 QA pairs over 16 public filings
from 10 companies. The mix is 56 factual, 24 calculation, 17 comparative, 13 table and 9 chart
questions. Every figure is traced to a quoted passage in its source document (209/209 verified
against the primary documents) and every derived number has an arithmetic check.
Re-checking found errors in 9 of the original 22 samples, and those have been corrected.
-
Scoring: RAGAS and an LLM-as-judge numeric scorer (needs an OpenAI key), plus two
deterministic checks that need no LLM: whether the answer states the expected values, and
whether it cites the expected source document.
-
Answer quality: run on all 119 questions with gemini-3.8-flash over top-5 hybrid
context, with deterministic scoring:
| Metric |
All 119 |
When retrieval found the supporting text (63) |
| Asked-for figure stated |
0.68 |
0.84 |
| Cites an expected document |
0.75 |
0.95 |
| Abstained (“not available”) |
0.35 |
0.16 |
| Fabricated numbers found |
0 |
0 |
Six answers stated numbers absent from their context, and all six were correct derivations.
Most misses are honest abstentions caused by retrieval. Calculation questions are weakest:
the model abstained on 15 of 24. Details are in
BENCHMARK_RESULTS.md.
pytest tests/unit/test_golden_dataset.py # dataset integrity, offline
python -m evals.retrieval_benchmark # retrieval, offline
python -m evals.answer_eval --rpm 5 # retrieval + generation, needs an LLM key
Latency and Cost
Not yet measured in a reproducible run. Earlier latency, load-test and cost-per-query charts
had no committed run outputs, so they are no longer cited. They are kept, with a provenance
note, in BENCHMARK_RESULTS.md section 4.
The only measured latency so far is generation alone, from the answer eval: p50 3.0 s,
p95 5.0 s with gemini-3.8-flash. tests/load/locustfile.py
and scripts/benchmark.py can measure them against a running deployment. The SLO targets
(p99 < 8 s, < 1% errors) are configured in src/rag_system/config.py and the alerting rules.
SLO Alerting
Design diagram, not live data: it illustrates how the configured alert rules behave and is
not taken from a running deployment. Multi-window multi-burn-rate alerting from the Google
SRE Workbook, with four alert tiers (14.4×, 6×, 3×, 1× burn rate) routing to PagerDuty/OpsGenie.
Architecture Layers
| Layer |
Component |
Technology |
| Parsing |
PDF text + tables |
Unstructured.io / Docling / Marker |
| Vision |
Chart + graph extraction |
GPT-4o / Gemini 2.5 Flash / Qwen2-VL / Local vLLM (fallback chain) |
| Layout |
Semantic grouping |
Table-caption pairing, multi-page merge, HTML wrapping |
| Embedding |
Dense vectors + cache |
OpenAI / Voyage / Cohere / local BAAI/bge, Redis cached |
| Indexing |
Vector store |
DeepLake / pgvector / Qdrant |
| Retrieval |
Hybrid (dense + BM25) |
Reciprocal Rank Fusion, k=60 |
| Reranking |
Cross-encoder |
ms-marco-MiniLM / Cohere Rerank v3 |
| Generation |
Cost-routed, multi-provider |
OpenAI / Gemini / Anthropic / Local vLLM |
| Guardrails |
Numeric grounding + PII |
Presidio + custom regex + restricted-interpreter PoT calculator |
| API |
REST + OpenAPI |
FastAPI + uvicorn |
| Observability |
Traces + metrics |
OpenTelemetry + Prometheus + Grafana |
| Security |
Auth + audit trail |
API key + SHA-256 tamper-evident audit log |
| Multi-tenancy |
Isolated namespaces |
Per-tenant vector partitions + quotas + rate limits |
| Deployment |
Container + K8s |
Docker + Helm + HPA + NetworkPolicy |
Key Features
Multimodal ingestion
- Vision LLM fallback chain: primary → secondary → local, never silently fails
- Layout-aware chunker: tables stay with their captions; multi-page tables merged
- ColPali visual retrieval (component only, not yet called by the query pipeline):
late-interaction MaxSim scoring on page images (no OCR)
- Delta detection: skip unchanged documents on re-ingest; version history for rollback
Retrieval
- Hybrid RRF (k=60): dense + BM25, equal-weight Reciprocal Rank Fusion. Measured: beats
both dense-only and BM25 at 1,000-char chunks (evidence hit@5 0.53 vs 0.46 and 0.42), and
ties BM25 at the default 4,000-char chunks (0.71). See Retrieval Quality
- Cross-encoder reranking: ms-marco-MiniLM-L-6-v2 or Cohere Rerank v3
- Query analyzer: intent classification → cost routing, entity extraction, query rewriting
- Semantic cache: a query whose embedding is close enough to a cached one (cosine ≥ 0.92)
is answered from cache, skipping retrieval and generation. The saving has not been measured
- Knowledge graph (optional,
ENABLE_KNOWLEDGE_GRAPH=true): LLM-extracted entities/relations
(COMPANY, METRIC, DATE; REPORTED_METRIC, FOR_PERIOD, …) and graph-augmented retrieval that walks
query entities to related metrics/periods and RRF-fuses the linked chunks with hybrid results.
The graph is in memory only (tutorial)
Quality assurance
- Numeric grounding guardrail: every number in the answer is verified against context
- Program-of-Thought calculator: the LLM writes the calculation; a restricted arithmetic
interpreter (no
exec(), no attribute access) evaluates it
- RAGAS evaluation: faithfulness, answer relevancy, context precision + LLM-as-judge
- Test suite: unit, integration, property-based (Hypothesis), API, chaos engineering. CI
runs it on every PR, with lint, mypy and a 70% coverage floor, and blocks on failure
Production operations
- OpenTelemetry: distributed traces with ingest/retrieve/generate spans
- 17 Prometheus metrics: latency, cost, hallucination score, citation coverage, cache hit rate
- Multi-window SLO alerting: Google SRE workbook burn-rate pattern, PagerDuty/OpsGenie
- Kubernetes: HPA 2–10 replicas, PodDisruptionBudget, NetworkPolicy, IRSA
- Terraform: full EKS + RDS(pgvector) + ElastiCache + S3 + KMS infrastructure-as-code
Try It Live
Interactive notebook: notebooks/quickstart.ipynb walks through the
full pipeline end-to-end — ingestion, hybrid retrieval, vision/chart queries, Program-of-Thought
calculations, guardrails, and cost tracking — using Tesla’s real Q3 2023 Investor Update as the
example document.
Hosted demo: Mattral/RAG-Financial-Multimodal
on Hugging Face Spaces (status RUNNING when checked on 2026-10-14). Bring your own OpenAI or
Gemini key. The Space defaults to gemini-2.5-flash, which returns 404 for newly created
Google API keys; pick a 3.x model or use OpenAI. On the free CPU tier the Space sleeps when
idle, so the first load can take a minute.
Its source is in spaces/rag-financial/. To run it locally:
cd spaces/rag-financial
pip install -r requirements.txt
python app.py
Bring your own OpenAI or Google Gemini API key (Gemini has a free tier). The Space mirrors
src/rag_system/'s hybrid retrieval + guardrail logic in a self-contained form with full
pipeline transparency — see exactly how each answer was retrieved and generated.
Repository Structure
src/rag_system/
├── api/ FastAPI app, routers (ingest/query/documents/tenants/feedback)
├── agentic/ LangGraph multi-step reasoning with self-correction loop
├── cli.py Typer CLI (ingest, query, evaluate, serve, health)
├── components/
│ ├── base.py ABCs for all pluggable components
│ ├── parser/ Unstructured, Docling, Marker adapters
│ ├── vision/ GPT-4o, Gemini, Qwen2-VL, LocalVLLM + fallback chain
│ ├── embedder/ OpenAI, Voyage, Cohere, local (BAAI/bge)
│ ├── vector_store/ DeepLake, pgvector, Qdrant
│ ├── retriever/ HybridRetriever (dense + BM25 + RRF)
│ ├── reranker/ CrossEncoder, Cohere, NoOp
│ ├── generator/ OpenAI, Gemini, Anthropic, LocalVLLM
│ ├── evaluator/ RAGAS + LLM-as-judge numeric scorer
│ ├── guardrails/ Numeric grounding, PII redaction, injection detection
│ ├── knowledge_graph.py Real LLM entity/relation extraction + graph traversal
│ ├── colpali_retriever.py Real MaxSim late-interaction visual retrieval
│ ├── pot_executor.py Program-of-Thought sandboxed calculator
│ ├── layout_parser.py Table-caption pairing, semantic chunking
│ ├── query_analyzer.py Intent classification, entity extraction
│ ├── version_manager.py Delta detection, point-in-time retrieval
│ └── connectors/ S3, Azure Blob, GCS
├── config.py Pydantic v2 BaseSettings, 12 nested sub-configs
├── pipeline/ RAGPipeline orchestrator (dependency injection)
├── sdk/ Python SDK (async + sync wrappers)
└── utils/ Telemetry, cost tracker, audit log, semantic cache, drift detector
terraform/ EKS + RDS(pgvector) + ElastiCache + S3 + IAM + KMS (IaC)
k8s/ Base manifests + Kustomize overlays (dev/prod)
helm/ Production Helm chart
spaces/ Standalone HuggingFace Space demos
└── rag-financial/ Gradio app — OpenAI + Gemini, hybrid retrieval, guardrails
Quick Reference
make setup # Install deps + copy .env
make dev # Start full stack with observability
make test # Run the test suite with coverage
make eval # Run RAGAS evaluation against golden dataset
make lint # Ruff lint
make typecheck # mypy
make docs # Serve MkDocs site
make query Q="What was Q3 revenue?"
Limitations
- Evaluation is partial. Dataset integrity, offline retrieval and retrieval-plus-generation
with one model are measured. The full ingest pipeline (unstructured parsing, reranker,
guardrails, Program-of-Thought), RAGAS scores, load latency and cost have not been measured
on a reproducible run yet (see BENCHMARK_RESULTS.md).
- Retrieval misses often. About 30% of golden questions get no answer-bearing chunk in the
top 5 at default settings, and multi-row tables are the weakest case when chunks are small.
- Small, single-author golden set. 119 questions on fiscal 2023, Tesla-heavy (40 of 119),
written by one person from the filings themselves. Their wording favors keyword search.
- Chart questions are text-checkable. The 9 chart questions use charts whose values also
appear in tables, so they do not isolate the vision path.
- Gemini judge not wired. The LLM-as-judge and RAGAS scorers call OpenAI. With only a Gemini
key, just the deterministic metrics are available.
- Calculations are often declined. Under the default “never estimate” system prompt the
model frequently refuses to compute derived figures (ratios, sums) unless Program-of-Thought
handles them.
- Page citations are unreliable when chunks lack page metadata. The model invents them.
- Thinking-token budget. On Gemini thinking models, reasoning tokens count against the
output limit.
LLM_CONFIG__THINKING_BUDGET_TOKENS (default 1024) now reserves them
separately; with it unset, answers can truncate.
- Retired model names. Google no longer serves
gemini-2.5-* to new API keys. Defaults
that still name them need a current model such as gemini-3.8-flash.
Documentation
License
MIT — see LICENSE. Built with care by @Mattral.
评论