chunking 开源项目推荐（837 个）

jiesutd /NCRFpp

NCRF++，一个神经序列标注工具包。易于用于任何序列标注任务（例如 NER、POS、分词）。它包括字符级 LSTM/CNN、词级 LSTM/CNN 以及 softmax/CRF 组件。

systemd /casync

内容寻址数据同步工具

isaacus-dev /semchunk

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

folbricht /desync

Alternative casync implementation

smooks /smooks

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

mirth /chonky

文本分块的全神经方法

ekimetrics /adaptive-chunking

Adaptive Chunking: automatically select the best chunking method per document for RAG. Accepted at LREC 2026.

microsoft /rag-experiment-accelerator

RAG实验加速器是一款多功能工具，旨在加快并促进使用Azure认知搜索和RAG模式进行实验和评估的过程。

zeroentropy-ai /zchunk

A new chunking strategy developed by ZeroEntropy for general semantic chunking using Llama-70B.

scub-france /docling-Studio

Visual document analysis studio powered by Docling — configure the extraction pipeline, inspect text, tables and bounding boxes in the browser, then chunk, embed and index into OpenSearch and Neo4j.

lazyFrogLOL /llmdocparser

A package for parsing PDFs and analyzing their content using LLMs.

drmingler /smart-llm-loader

smart-llm-loader 是一个轻量却强大的 Python 包，可将任意文档转换为可直接供 LLM 使用的分块。减少预处理烦恼的时间，把更多时间用于构建重要之事。从 RAG 系统到聊天机器人再到文档问答，SmartLLMLoader 承担繁重工作，让你专注于打造出色的 AI 应用。

26hzhang /neural_sequence_labeling

神经序列标注模型的 TensorFlow 实现，能够处理词性标注、组块分析、命名实体识别、标点恢复等序列标注任务。

GiovanniPasq /chunky

用于可靠 RAG 流水线的开源工具包：将 PDF 转换为 Markdown、清理文档、检查分块、比较分块策略，并为 LLM 应用丰富元数据。

bzsanti /oxidizePdf

Pure Rust PDF library for AI/RAG: structure-aware chunking, no ML, no C deps.

AKSarav /pdfstract

PDFStract - Extract, Chunking and Embedding Layer in Your RAG Pipeline - Available as CLI - WEBUI - API

jparkerweb /semantic-chunking

🍱 Semantically create chunks from large document for passing to LLM workflows

messkan /rag-chunk

一个 Python CLI，用于测试、基准评估并为你 Markdown 文档找到最佳 RAG 分块策略。

neondatabase /pgrag

Deprecated/unmaintained Postgres extensions to support end-to-end Retrieval-Augmented Generation (RAG) pipelines

safakatakancelik /TalkWithYourFiles

Upload your files and ask questions about them. Open-source RAG app with LangChain & Streamlit, built before ChatGPT had file uploads. Works with Anthropic & OpenRouter. Easy to extend. The newest release is a functioning tribute and future development ideas are in consideration.