pretrain 开源项目推荐（28 个）

brightmart /nlp_chinese_corpus

大规模中文自然语言处理语料 用于自然语言处理的大规模中文语料库

keyu-tian /SparK

[ICLR'23 Spotlight🔥] 首次成功在任何卷积网络上进行BERT/MAE风格的预训练；"Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"的Pytorch实现

InternScience /GraphGen

GraphGen：通过知识驱动的合成数据生成增强LLM的监督微调

CLUEbenchmark /CLUECorpus2020

大规模中文预训练语料库 100G 中文预训练语料

yangjianxin1 /Firefly-LLaMA2-Chinese

Firefly 中文 LLaMA-2 大模型，支持对 Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom 等大模型进行增量预训练

microsoft /UniVL

“UniVL：用于多模态理解与生成的统一视频与语言预训练模型”的官方实现

walkinglabs /modern-llm-notebook

一门关于现代 LLM 架构、训练和推理的实践课程，包含逐步的 PyTorch 实现和可运行的 notebook。

datawhalechina /code-your-own-llm

一份全栈式大语言模型参考指南，用最简洁的代码帮助你端到端定义模型从零训练到工程落地的每一个细节

THUNLP-AIPoet /BERT-CCPoem

BERT-CCPoem 是一个专门面向中国古典诗词的基于 BERT 的预训练模型

xcfcode /What-I-Have-Read

论文列表、笔记和幻灯片，聚焦 NLP。关于文本摘要，请参阅 https://github.com/xcfcode/Summarization-Papers

thunlp /RE-Context-or-Names

基于 Bert 的模型（BERT、MTB、CP）用于关系抽取。

wxhcore /bumblecore

An LLM training framework built from the ground up, featuring a custom BumbleBee architecture and end-to-end support for multiple open-source models across Pretraining → SFT → RLHF/DPO.

huzongxiang /MatDGL

MatDGL is a neural network package that allows researchers to train custom models for crystal modeling tasks. It aims to accelerate the research and application of material science.

SalesforceAIResearch /pretrain-time-series-cloudops

Official code repository for the paper "Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain"

CoinCheung /MFM

code for paper "Masked Frequency Modeling for Self-Supervised Visual Pre-Training" (https://arxiv.org/pdf/2206.07706.pdf)

nancheng58 /SSL4SR

[CCIR 2023] Self-supervised learning for Sequential Recommender Systems

bayartsogt-ya /albert-mongolian

ALBERT trained on Mongolian text corpus

KennethanCeyer /diy-generative-ai-lm

Make your Generative AI LM model from the scratch (Including pretraining / SFT with LoRA)

yongzhuo /MacroGPT-Pretrain

macrogpt大模型全量预训练(1b3,32层), 多卡deepspeed/单卡adafactor

mrzjy /hoyo_public_wiki_parser

Parsing Hoyoverse game text corpus from public wikipedia