semantic-chunking 开源项目推荐（82 个）

isaacus-dev /semchunk

一个快速、轻量且易于使用的 Python 库，用于将文本拆分为语义上有意义的块。

mirth /chonky

文本分块的全神经方法

jparkerweb /semantic-chunking

🍱 Semantically create chunks from large document for passing to LLM workflows

mburaksayici /RAG-Boilerplate

RAG boilerplate with semantic/propositional chunking, hybrid search (BM25 + dense), LLM reranking, query enhancement agents, CrewAI orchestration, Qdrant vector search, Redis/Mongo sessioning, Celery ingestion pipeline, Gradio UI, and an evaluation suite (Hit-Rate, MRR, hybrid configs).

zircote /rlm-rs

Rust CLI implementing the Recursive Language Model (RLM) pattern for Claude Code. Process documents 100x larger than context windows through intelligent chunking, SQLite persistence, and recursive sub-LLM orchestration.

poloniki /quint

Turn speech into semantic paragraphs in real time, on a CPU (no GPU) — single-pass streaming chunker + Moonshine STT + a live mic demo. Plus a transcribe/chunk/summarize API.

zhiweio /EagleRAG

Search knowledge by what documents mean and how they look — not one or the other.

jparkerweb /llm-distillery

🍶 llm-distillery ⇢ use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.

prajwal10001 /semantic-chunker-langchain

Token-aware, LangChain-compatible semantic chunker with PDF, markdown, and layout support

ThanhHung2112 /Semantic_chunking

Semantic Chunking is a Python library for segmenting text into meaningful chunks using embeddings from Sentence Transformers.

bazilicum /axonode-chunker

Advanced semantic text chunking with custom structural markers, whole-text coherence preservation, and flexible token management. Features async processing, LangChain integration, and dynamic drift detection. Ideal for RAG systems, augmented text processing, and domain-specific document analysis.

wigtn /wigtnOCR-v1

Qwen3-VL-30B → 2B distillation for document parsing; the production parser in RCPS (EMNLP 2026).

darkzard05 /rag-system-ollama

Advanced local-first RAG system powered by Ollama and LangGraph. Optimized for high-performance sLLM orchestration featuring adaptive intent routing, semantic chunking, intelligent hybrid search (FAISS + BM25), and real-time thought streaming. Includes integrated PDF analysis and secure vector caching.

richie-rich90454 /training-generator

Local-first desktop application for converting documents into AI training datasets. Generate instruction, conversation, and fine-tuning datasets from PDFs, DOCX, Markdown, HTML, and text using Ollama or cloud LLMs.

Jeevav62 /chunking-techniques

A practical guide to 6 document chunking strategies for RAG and LLM applications — Document, Fixed-Size, Recursive, Sentence, Semantic, and Agentic chunking with working code and plain-English explanations.

omega-sus67 /rag_python

A RAG pipeline whose agent cannot answer from memory: retrieval is enforced, and every citation is verified against the database. Built from scratch in Python (FastAPI, Celery, pgvector HNSW) with a measured retrieval benchmark against a fixed-size baseline.

4thel00z /konan

Blazingly fast text chunkers in Rust with pythonic bindings. Rayon-parallel, real async, char-accurate offsets, semantic chunking

wenisch-tech /ContextCrate

ContextCrate is an open-source, self-hosted retrieval-augmented generation platform. It ingests websites and Git repositories, builds searchable context, and delivers grounded answers with citations you can inspect.

anujmumbaikar /Optimizing-RAG-Pipeline-Hybrid-Search-RRF-Fusion-Re-Ranking

A high-performance Retrieval-Augmented Generation pipeline for technical Q&A workloads. Combines hybrid retrieval (dense + BM25), query expansion, Reciprocal Rank Fusion (RRF), and cross-encoder re-ranking to improve retrieval precision and answer grounding. Evaluated with Ragas, showing measurable gains in context recall and faithfulness.

njyeung /go-semantic-chunking

Sementic chunking algorithm in (mostly) Go