raglite

🥤 RAGLite

RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL.

Features

Configurable

• 🧠 Choose any LLM provider with LiteLLM, including local llama-cpp-python models
• 💾 Choose either DuckDB or PostgreSQL as a keyword & vector search database
• 🥇 Choose any reranker with rerankers, including multilingual FlashRank as the default

Fast and permissive

• ❤️ Only lightweight and permissive open source dependencies (e.g., no PyTorch or LangChain)
• 🚀 Acceleration with Metal on macOS, and CUDA on Linux and Windows

Unhobbled

• 📖 PDF to Markdown conversion on top of pdftext and pypdfium2
• 🧬 Multi-vector chunk embedding with late chunking and contextual chunk headings
• ✏️ Optimal sentence splitting with wtpsplit-lite by solving a binary integer programming problem
• ✂️ Optimal semantic chunking by solving a binary integer programming problem
• 🔍 Hybrid search with the database’s native keyword & vector search (FTS+VSS; tsvector+pgvector)
• 💭 Adaptive retrieval where the LLM decides whether to and what to retrieve based on the query
• 💰 Improved cost and latency with a prompt caching-aware message array structure
• 🍰 Improved output quality with Anthropic’s long-context prompt format
• 🌀 Optimal closed-form linear query adapter by solving an orthogonal Procrustes problem

Extensible

• 🔌 A built-in Model Context Protocol (MCP) server that any MCP client like Claude desktop can connect with
• 💬 Optional customizable ChatGPT-like frontend for web, Slack, and Teams with Chainlit
• ✍️ Optional conversion of any input document to Markdown with Pandoc
• 🔎 Optional high-quality document processing with Mistral OCR for PDFs, images, DOCX, and PPTX with automatic image descriptions
• ✅ Optional evaluation of retrieval and generation performance with Ragas

Installing

# Configure which llama-cpp-python precompiled binary to install (⚠️ not every combination is available):
LLAMA_CPP_PYTHON_VERSION=0.3.9
PYTHON_VERSION=310|311|312
ACCELERATOR=metal|cu121|cu122|cu123|cu124
PLATFORM=macosx_11_0_arm64|linux_x86_64|win_amd64

# Install llama-cpp-python:
pip install "https://github.com/abetlen/llama-cpp-python/releases/download/v$LLAMA_CPP_PYTHON_VERSION-$ACCELERATOR/llama_cpp_python-$LLAMA_CPP_PYTHON_VERSION-cp$PYTHON_VERSION-cp$PYTHON_VERSION-$PLATFORM.whl"

Install RAGLite with:

pip install raglite

To add support for a customizable ChatGPT-like frontend, use the chainlit extra:

pip install raglite[chainlit]

To add support for filetypes other than PDF, use the pandoc extra:

pip install raglite[pandoc]

To add support for high-quality document processing with Mistral OCR, install mistralai:

pip install mistralai

To add support for evaluation, use the ragas extra:

pip install raglite[ragas]

Using

Overview

1. Configuring RAGLite
2. Inserting documents
3. Retrieval-Augmented Generation (RAG)
4. Computing and using an optimal query adapter
5. Evaluation of retrieval and generation
6. Running a Model Context Protocol (MCP) server
7. Serving a customizable ChatGPT-like frontend

1. Configuring RAGLite

First, configure RAGLite with your preferred DuckDB or PostgreSQL database and any LLM supported by LiteLLM:

from raglite import RAGLiteConfig

# Example 'remote' config with a PostgreSQL database and an OpenAI LLM:
my_config = RAGLiteConfig(
    db_url="postgresql://my_username:my_password@my_host:5432/my_database",
    llm="gpt-4o-mini",  # Or any LLM supported by LiteLLM
    embedder="text-embedding-3-large",  # Or any embedder supported by LiteLLM
)

# Example 'local' config with a DuckDB database and a llama.cpp LLM:
my_config = RAGLiteConfig(
    db_url="duckdb:///raglite.db",
    llm="llama-cpp-python/unsloth/Qwen3-8B-GGUF/*Q4_K_M.gguf@8192",
    embedder="llama-cpp-python/lm-kit/bge-m3-gguf/*F16.gguf@512", # More than 512 tokens degrades bge-m3's performance
)

You can also configure any reranker supported by rerankers:

from rerankers import Reranker

# Example remote API-based reranker:
my_config = RAGLiteConfig(
    db_url="postgresql://my_username:my_password@my_host:5432/my_database"
    reranker=Reranker("rerank-v3.5", model_type="cohere", api_key=COHERE_API_KEY, verbose=0)  # Multilingual
)

# Example local cross-encoder reranker per language (this is the default):
my_config = RAGLiteConfig(
    db_url="duckdb:///raglite.db",
    reranker={
        "en": Reranker("ms-marco-MiniLM-L-12-v2", model_type="flashrank", verbose=0),  # English
        "other": Reranker("ms-marco-MultiBERT-L-12", model_type="flashrank", verbose=0),  # Other languages
    }
)

Self-query is also supported, allowing the LLM to automatically generate and apply metadata filters to refine search results based on the user’s input. To enable self-query, set self_query=True in your RAGLiteConfig:

my_config = RAGLiteConfig(
    db_url="duckdb:///raglite.db",
    llm="gpt-4o-mini",
    embedder="text-embedding-3-large",
    self_query=True,  # Enable self-query
)

2. Inserting documents

from raglite import RAGLiteConfig, MistralOCRConfig

my_config = RAGLiteConfig(
    document_processor=MistralOCRConfig(
        include_image_descriptions=True,  # Describe images, charts, and diagrams as text
        image_types=frozenset({"chart", "diagram", "photo", "table", "logo", "icon"}),  # Custom image categories
        exclude_image_types=frozenset({"logo", "icon"}),  # Filter out specific types from the output
    ),
)

Next, insert some documents into the database. RAGLite will take care of the conversion to Markdown, optimal level 4 semantic chunking, and multi-vector embedding with late chunking:

# Insert documents given their file path
from pathlib import Path
from raglite import Document, insert_documents

documents = [
    Document.from_path(Path("On the Measure of Intelligence.pdf")),
    Document.from_path(Path("Special Relativity.pdf")),
]
insert_documents(documents, config=my_config)

# Insert documents given their text/plain or text/markdown content
content = """
# ON THE ELECTRODYNAMICS OF MOVING BODIES
## By A. EINSTEIN  June 30, 1905
It is known that Maxwell...
"""
documents = [
    Document.from_text(content, author="Einstein", topic="physics", year=1905)
]
insert_documents(documents, config=my_config)

[!TIP] 📝 Documents can include metadata by passing keyword arguments to Document.from_text() or Document.from_path(). This metadata can later be used for filtering during retrieval. For list values, metadata is stored as-is (e.g. domain=["open", "music"]).

You may also want to expand the document metadata before insertion:

from typing import Annotated, Literal
from pydantic import Field
from raglite import expand_document_metadata

# Expand the documents' metadata.
metadata_fields = {
    "title": Annotated[str, Field(..., description="Document title.")],
    "author": Annotated[str, Field(..., description="Primary author.")],
    "topics": Annotated[list[Literal["A", "B", "C"]], Field(..., description="Key themes.")],
}
documents = list(expand_document_metadata(documents, metadata_fields, config=my_config))

# Insert documents given their text/plain or text/markdown content
insert_documents(documents, config=my_config)

3. Retrieval-Augmented Generation (RAG)

3.1 Adaptive RAG

Now you can run an adaptive RAG pipeline that consists of adding the user prompt to the message history and streaming the LLM response:

from raglite import rag

# Create a user message
messages = []  # Or start with an existing message history
messages.append({
    "role": "user",
    "content": "How is intelligence measured?"
})

# Adaptively decide whether to retrieve and then stream the response
chunk_spans = []
stream = rag(messages, on_retrieval=lambda x: chunk_spans.extend(x), config=my_config)
for update in stream:
    print(update, end="")

# Access the documents referenced in the RAG context
documents = [chunk_span.document for chunk_span in chunk_spans]

The LLM will adaptively decide whether to retrieve information based on the complexity of the user prompt. If retrieval is necessary, the LLM generates the search query and RAGLite applies hybrid search and reranking to retrieve the most relevant chunk spans (each of which is a list of consecutive chunks). The retrieval results are sent to the on_retrieval callback and are appended to the message history as a tool output. Finally, the assistant response is streamed and appended to the message history.

3.2 Programmable RAG

If you need manual control over the RAG pipeline, you can run a basic but powerful pipeline that consists of retrieving the most relevant chunk spans with hybrid search and reranking, converting the user prompt to a RAG instruction and appending it to the message history, and finally generating the RAG response:

from raglite import add_context, rag, retrieve_context, vector_search

# Choose a search method
from dataclasses import replace
my_config = replace(my_config, search_method=vector_search)  # Or `hybrid_search`, `search_and_rerank_chunks`, ...

# Retrieve relevant chunk spans with the configured search method
user_prompt = "How is intelligence measured?"
chunk_spans = retrieve_context(
    query=user_prompt, 
    num_chunks=5, 
    metadata_filter={"author": "Einstein"},  # Optional: filter by metadata
    config=my_config
)

# Append a RAG instruction based on the user prompt and context to the message history
messages = []  # Or start with an existing message history
messages.append(add_context(user_prompt=user_prompt, context=chunk_spans, config=my_config))

# Stream the RAG response and append it to the message history
stream = rag(messages, config=my_config)
for update in stream:
    print(update, end="")

# Access the documents referenced in the RAG context
documents = [chunk_span.document for chunk_span in chunk_spans]

RAGLite also offers more advanced control over the individual steps of a full RAG pipeline:

1. Searching for relevant chunks with keyword, vector, or hybrid search
2. Retrieving the chunks from the database
3. Reranking the chunks and selecting the top 5 results
4. Extending the chunks with their neighbors and grouping them into chunk spans
5. Converting the user prompt to a RAG instruction and appending it to the message history
6. Streaming an LLM response to the message history
7. Accessing the cited documents from the chunk spans

A full RAG pipeline is straightforward to implement with RAGLite:

# Search for chunks
from raglite import hybrid_search, keyword_search, vector_search

user_prompt = "How is intelligence measured?"
chunk_ids_vector, _ = vector_search(user_prompt, num_results=20, config=my_config)
chunk_ids_keyword, _ = keyword_search(user_prompt, num_results=20, config=my_config)
chunk_ids_hybrid, _ = hybrid_search(
    user_prompt, num_results=20, metadata_filter={"topic": "physics"}, config=my_config
)  # Filter results to only include chunks from documents with topic="physics" (works with any search method)

# Multi-value filter in one field uses OR semantics:
chunk_ids_or, _ = hybrid_search(
    user_prompt,
    num_results=20,
    metadata_filter={"domain": ["open", "music"]},
    config=my_config,
)  # Returns chunks where domain includes "open" OR "music".

# Retrieve chunks
from raglite import retrieve_chunks

chunks_hybrid = retrieve_chunks(chunk_ids_hybrid, config=my_config)

# Rerank chunks and keep the top 5 (optional, but recommended)
from raglite import rerank_chunks

chunks_reranked = rerank_chunks(user_prompt, chunks_hybrid, config=my_config)
chunks_reranked = chunks_reranked[:5]

# Extend chunks with their neighbors and group them into chunk spans
from raglite import retrieve_chunk_spans

chunk_spans = retrieve_chunk_spans(chunks_reranked, config=my_config)

# Append a RAG instruction based on the user prompt and context to the message history
from raglite import add_context

messages = []  # Or start with an existing message history
messages.append(add_context(user_prompt=user_prompt, context=chunk_spans, config=my_config))

# Stream the RAG response and append it to the message history
from raglite import rag

stream = rag(messages, config=my_config)
for update in stream:
    print(update, end="")

# Access the documents referenced in the RAG context
documents = [chunk_span.document for chunk_span in chunk_spans]

4. Computing and using an optimal query adapter

RAGLite can compute and apply an optimal closed-form query adapter to the prompt embedding to improve the output quality of RAG. To benefit from this, first generate a set of evals with insert_evals and then compute and store the optimal query adapter with update_query_adapter:

# Improve RAG with an optimal query adapter
from raglite import insert_evals, update_query_adapter

insert_evals(num_evals=100, config=my_config)
update_query_adapter(config=my_config)  # From here, every vector search will use the query adapter

5. Evaluation of retrieval and generation

If you installed the ragas extra, you can use RAGLite to answer the evals and then evaluate the quality of both the retrieval and generation steps of RAG using Ragas:

# Evaluate retrieval and generation
from raglite import answer_evals, evaluate, insert_evals

insert_evals(num_evals=100, config=my_config)
answered_evals_df = answer_evals(num_evals=10, config=my_config)
evaluation_df = evaluate(answered_evals_df, config=my_config)

6. Running a Model Context Protocol (MCP) server

RAGLite comes with an MCP server implemented with FastMCP that exposes a search_knowledge_base tool. To use the server:

1. Install Claude desktop
2. Install uv so that Claude desktop can start the server
3. Configure Claude desktop to use uv to start the MCP server with:

raglite \
    --db-url duckdb:///raglite.db \
    --llm llama-cpp-python/unsloth/Qwen3-4B-GGUF/*Q4_K_M.gguf@8192 \
    --embedder llama-cpp-python/lm-kit/bge-m3-gguf/*F16.gguf@512 \
    mcp install

To use an API-based LLM, make sure to include your credentials in a .env file or supply them inline:

export OPENAI_API_KEY=sk-...
raglite \
    --llm gpt-4o-mini \
    --embedder text-embedding-3-large \
    mcp install

Now, when you start Claude desktop you should see a 🔨 icon at the bottom right of your prompt indicating that the Claude has successfully connected with the MCP server.

When relevant, Claude will suggest to use the search_knowledge_base tool that the MCP server provides. You can also explicitly ask Claude to search the knowledge base if you want to be certain that it does.

7. Serving a customizable ChatGPT-like frontend

If you installed the chainlit extra, you can serve a customizable ChatGPT-like frontend with:

raglite chainlit

The application is also deployable to web, Slack, and Teams.

You can specify the database URL, LLM, and embedder directly in the Chainlit frontend, or with the CLI as follows:

raglite \
    --db-url duckdb:///raglite.db \
    --llm llama-cpp-python/unsloth/Qwen3-4B-GGUF/*Q4_K_M.gguf@8192 \
    --embedder llama-cpp-python/lm-kit/bge-m3-gguf/*F16.gguf@512 \
    chainlit

To use an API-based LLM, make sure to include your credentials in a .env file or supply them inline:

OPENAI_API_KEY=sk-... raglite --llm gpt-4o-mini --embedder text-embedding-3-large chainlit

Contributing

Star History