bpe 开源项目推荐（367 个）

rsennrich /subword-nmt

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

VKCOM /YouTokenToMe

Unsupervised text tokenizer focused on computational efficiency

niieani /gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

zurawiki /tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

OpenNMT /Tokenizer

快速且可定制的文本分词库，支持 BPE 和 SentencePiece

ImadSaddik /Train_Your_Language_Model_Course

Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.

myylogic /cevahir-ai

Full-stack open-source AI engine for building language models — tokenizer training, transformer architecture, cognitive reasoning and chat pipeline.

Kyubyong /nlp_made_easy

Explains nlp building blocks in a simple manner.

soaxelbrooke /python-bpe

Byte Pair Encoding for Python!

akretion /nfelib

nfelib - 用于读取和生成 NF-e、NFS-e nacional、CT-e、MDF-e、BP-e XML 的 Python 绑定

gautierdag /bpeasy

Fast bare-bones BPE for modern tokenizer training

fangpin /llm-from-scratch

Build LLM from scratch

actual-computer /toks

精确的分词器 ID，速度达到硬件允许的极限

sefineh-ai /Amharic-Tokenizer

Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.

tryAGI /Tiktoken

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

samber /go-gpt-3-encoder

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Systemcluster /kitoken

面向语言模型的快速分词器，兼容 SentencePiece、Tokenizers、Tiktoken 等。支持在 JavaScript、Python 和 Rust 中进行 BPE、Unigram 和 WordPiece 分词。

faizann24 /phishytics-machine-learning-for-phishing

Machine Learning for Phishing Website Detection

gillverd /accelerando-8192

Accelerando-8192 — an 8,192-token alien glyph alphabet, interactive encoder, and complete visual atlas.

jiesutd /SubwordEncoding-CWS

Subword Encoding in Lattice LSTM for Chinese Word Segmentation