pretrain 开源项目推荐（28 个）

brightmart /nlp_chinese_corpus

大规模中文自然语言处理语料 用于自然语言处理的大规模中文语料库

keyu-tian /SparK

[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

InternScience /GraphGen

GraphGen：通过知识驱动的合成数据生成增强LLM的监督微调

CLUEbenchmark /CLUECorpus2020

Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料

yangjianxin1 /Firefly-LLaMA2-Chinese

Firefly 中文 LLaMA-2 大模型，支持对 Baichuan2、Llama2、Llama、Falcon、Qwen、Baichuan、InternLM、Bloom 等大模型进行增量预训练

microsoft /UniVL

An official implementation for " UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation"

walkinglabs /modern-llm-notebook

A hands-on course on modern LLM architectures, training, and inference, with step-by-step PyTorch implementations and runnable notebooks.

datawhalechina /code-your-own-llm

一份全栈式大语言模型参考指南，用最简洁的代码帮助你端到端定义模型从零训练到工程落地的每一个细节

THUNLP-AIPoet /BERT-CCPoem

BERT-CCPoem is an BERT-based pre-trained model particularly for Chinese classical poetry

xcfcode /What-I-Have-Read

Paper Lists, Notes and Slides, Focus on NLP. For summarization, please refer to https://github.com/xcfcode/Summarization-Papers

thunlp /RE-Context-or-Names

Bert-based models(BERT, MTB, CP) for relation extraction.

wxhcore /bumblecore

An LLM training framework built from the ground up, featuring a custom BumbleBee architecture and end-to-end support for multiple open-source models across Pretraining → SFT → RLHF/DPO.

huzongxiang /MatDGL

MatDGL is a neural network package that allows researchers to train custom models for crystal modeling tasks. It aims to accelerate the research and application of material science.

SalesforceAIResearch /pretrain-time-series-cloudops

Official code repository for the paper "Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain"

CoinCheung /MFM

code for paper "Masked Frequency Modeling for Self-Supervised Visual Pre-Training" (https://arxiv.org/pdf/2206.07706.pdf)

nancheng58 /SSL4SR

[CCIR 2023] Self-supervised learning for Sequential Recommender Systems

bayartsogt-ya /albert-mongolian

ALBERT trained on Mongolian text corpus

KennethanCeyer /diy-generative-ai-lm

Make your Generative AI LM model from the scratch (Including pretraining / SFT with LoRA)

yongzhuo /MacroGPT-Pretrain

macrogpt大模型全量预训练(1b3,32层), 多卡deepspeed/单卡adafactor

mrzjy /hoyo_public_wiki_parser

Parsing Hoyoverse game text corpus from public wikipedia