length-extrapolation 开源项目推荐（13 个）

Xnhyacinth /Awesome-LLM-Long-Context-Modeling

必读基于LLM的长上下文建模的论文和博客🔥

OpenMOSS /LongLLaDA

Methods and code for extending the context length of diffusion language models

OpenMOSS /rope_pp

An imaginary extension of rotary position embeddings for long-context language models

AlgonetLabs /Cable

Context-aware Biases for Length Extrapolation

OpenMOSS /ReAttention

A training-free method for extending the maximum context length of language models

zifei-bai /Sliding-Critical-Band-in-RoPE-based-Length-Extrapolation

该项目在 GitHub 上没有填写描述

zekun-shi /freesolo

Subquadratic sparse attention that zero-shot length extrapolation to 256k+ tokens, with fast CUDA kernels. arXiv:2610.17191

duongtrongnguyen123 /recurrent-depth-ttc

Controlled experiments on recurrent-depth (looped) transformers — length extrapolation, adaptive test-time compute, and honest sub-1B negatives.

sarimahsan /pos-encoding-ablation

A modular, config-driven PyTorch framework for conducting empirical positional encoding ablation experiments on decoder-only transformers (~51M parameters) using Google Colab T4 GPUs

Royer-Research-Labs /Selective-Attention-Study

Selective Attention trained like Dilution-Attention: no position signal, 209M parameters, 16K context, with a fused Triton kernel.

Royer-Research-Labs /Niah

Likelihood-based Needle-In-A-Haystack eval for small and from-scratch language models: multiple-choice scoring with a control prior (no generation needed), a hard mode with natural filler and keyed distractors, depth × length heatmaps, during-training eval and an LM Evaluation Harness task.

DT-Foss /O1

A constant-memory recurrent stream coupled to a growing .causal knowledge index, with a measured capacity threshold in the gated readout. Builds on GSSM.

DT-Foss /o1-state

The O(1)-state organism: constant-memory unbounded streaming, surprise-gated plasticity, holographic memory through silence, dosed sleep, a runtime knowledge index — and live cross-architecture migration. Seven disclosed foundations, 38 pre-registered predictions, falsifications kept.