OpenAI Model Craft Challenge: Parameter Golf is a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte).
This challenge is heavily inspired by the NanoGPT Speedrunning challenge, where participants compete to train a model that reaches 3.28 FineWeb validation loss as quickly as possible. We’re excited to see how optimizing for a parameter-constrained setting pushes people toward unique architectures (test-time compute, aggressive parameter tying, depth recurrence, low-rank training, …), compression schemes (low precision, QAT, bitnets, novel tokenizers, …), and other creative submissions (test-time training, long context, megakernels …).
If you’re familiar with neural scaling laws, you can consider this challenge a form of L(N) optimization, where the objective is to optimize the lowest loss given a fixed number of parameters (N) unconstrained by data, compute, steps, or architecture. Challenges like the NanoGPT Speedrun, which optimizes for a form of L(T) (~lowest time given constrained loss) or the NanoGPT Slowrun, which optimizes for L(D) (lowest loss given constrained dataset size), can be thought of as equivalent challenges in this family.
Ideally, we’d allow for submissions to use arbitrary computational resources. But in order to make the challenge not inaccessibly expensive, we’re limiting leaderboard submissions to 10 minutes on 8xH100s. However, we’d still love to see submissions that don’t meet the compute limitation requirements in our ‘Non-record Submissions’ section: We’re excited to see people push the infinite frontier of parameter limited performance as well.
We also know compute is expensive, so OpenAI is sponsoring $1,000,000 in compute credits to help people get started training their models. To request a compute grant, use this form: Request a Compute Grant.
When requesting compute, please make sure you choose the appropriate level, write sufficient justification, and submit with an email tied to a OpenAI / ChatGPT account.
If you enjoy solving very difficult technical problems, please introduce yourself via the Challenge Participant Form. It helps us attribute challenge submissions and reach out about opportunities with OpenAI. Completing the form is not required to participate.
Many researchers at OpenAI first distinguished themselves through elite mathematics and programming competitions. The Model Craft Challenge is designed in that spirit: testing the ability to tackle unfamiliar problems with creativity and rigor, qualities we believe are essential for frontier AI research.
In June, we plan to hire a small cohort of early-career researchers, targeting current undergraduate students and recent graduates, including Olympiad medalists and elite competitors. For exceptional participants, the challenge may also serve as a way to stand out to OpenAI researchers and recruiters.
The challenge runs from March 18th to April 30th.
Happy training!
Leaderboard
| Run |
Score |
Author |
Summary |
Date |
Info |
| Calib32 Token-Only N-gram + AsymLogit Stack |
1.0565 |
codemath3000 |
On PR #2135: pre-cutoff PR #2130 architecture rerun on clean canonical CaseOps data with GPTQ_CALIBRATION_BATCHES=32; 3-seed mean 1.05651 under grace policy (p=0.014 vs PR #2014) |
2026-05-01 |
info |
| Progressive Context Growth + Short-Doc Score-First TTT |
1.0576 |
simonbissonnette |
On PR #2014: PR #1855/#1953 CaseOps stack with progressive context growth to 3k plus short-doc score-first TTT on the AWQ-lite/AsymLogit lineage; 3-seed mean 1.05759 (p=0.011 vs PR #1953) |
2026-04-30 |
info |
| Long-Context No-Q/V TTT + QK-Gain 5.25 |
1.0586 |
andrewbaggio1 |
On PR #1953: PR #1945 V21 base with 2560 eval/TTT context, no-Q/V TTT mask, TTT LR 0.75, and QK_GAIN_INIT=5.25; 3-seed mean 1.05855 (p=0.063 vs PR #1945 V21 v2) |
2026-04-30 |
info |
| AWQ-Lite GPTQ + AsymLogit on PR1855 Stack |
1.0594 |
alertcat |
On PR #1945 commit 70067534: PR #1855 stack plus PR #1908 AWQ-lite mixed GPTQ and PR #1923 AsymLogit; V21 v2 3-seed mean 1.05943 after strict seed-42 rerun (p=0.034 vs PR #1855) |
2026-04-29 |
info, commit |
| BOS-Fixed SmearGate + LQER + SparseAttnGate + 9-Hparam Stack |
1.0611 |
codemath3000 |
On PR #1855: BOS-fixed #1797-derived stack with LQER, PR #1787 SparseAttnGate/PolarNS/FusedCE base, per-group lrzip compression, and 9 greedy hyperparameter overrides; submitted 3-seed mean 1.06108 with broader reproduction support (p=0.188 vs PR #1868 latest rerun) |
2026-04-27 |
info, repro |
| BOS-Fixed SmearGate + LQER Asymmetric + PR1787 SparseAttn + Phased TTT |
1.0614 |
aquariouseworkman |
On PR #1851 with 3-seed compliance-rerun support from PR #1868: BOS-boundary fix from PR #1851 applied to dexhunter’s PR #1797 SmearGate + LQER stack, using the PR #1787 SparseAttnGate/PolarNS/FusedCE base plus CaseOps and phased score-first TTT |
2026-04-27 |
info, 3-seed |
| PR1736 + PolarNS + MIN_LR + SparseAttnGate + FusedCE + Warm-A TTT |
1.0634 |
nprime06 |
On PR #1787: PR #1736 CaseOps stack plus Polar Express Newton-Schulz coefficients, MIN_LR=0.1, SparseAttnGate, fused softcapped CE, and PR #1767-style warm-start-A TTT |
2026-04-23 |
info |
| CaseOps + MLPClip12 + SmearGate/LoRA-TTT |
1.0645 |
dexhunter |
On PR #1769: CaseOps stack with SmearGate/LoRA-TTT refinements and MLPClip12; 5-seed mean improves the accepted CaseOps frontier (p=0.063 vs #1736) |
2026-04-22 |
info |
| SP8192 + CaseOps + GatedAttn + QuantGate + Loop45 + Phased TTT |
1.0655 |
dexhunter |
On PR #1736: adopts romeerp’s lossless CaseOps transform from PR #1729 with byte-sidecar BPB accounting, then adds gated attention and quant-gate scaling on the PR #1530 SP8192 phased-TTT stack |
2026-04-19 |
info |
| CaseOps Tokenizer + Tapered WD + Phased TTT |
1.0678 |
romeerp |
On PR #1729: lossless CaseOps bijective case transform with validation byte sidecars, plus mild late Muon weight-decay taper on the PR #1626 legal phased-TTT stack |
2026-04-19 |
info |
| SmearGate + Attention Output Gate + Legal TTT |
1.0714 |
MarioPaerle |
On PR #1667: SmearGate, attention output gate, depth recurrence, parallel residuals, QK-Gain 5.25, quantization, and score-first TTT |
2026-04-16 |
info |
| VarLen Attention + Fused MLP + Multi-Phase Global SGD TTT |
1.0719 |
dexhunter |
On PR #1626: VarLen attention, fused MLP, multi-phase global SGD TTT, trimmed GPTQ, MLR 0.026, int7 embeddings, and adaptive clip |
2026-04-14 |
info |
| VarLenAttn + PhasingTTT |
1.0728 |
romeerp |
On PR #1610: #1530-style VarLen/fused stack plus phased TTT over already-scored validation chunks |
2026-04-13 |
info |
| VarLen Attention + Fused MLP + Doc-Independent Legal TTT |
1.0734 |
samacqua |
On PR #1530: variable-length FA3 attention, fused Triton MLP, grouped small-parameter all-reduces, and doc-independent score-first LoRA TTT |
2026-04-11 |
info |
| Improved Parallel Residuals + CUTLASS EVT + Legal TTT |
1.0758 |
msisovic |
On PR #1529: PR #1523 SP8192 baseline with fuller two-lane parallel residual routing, PARALLEL_RESIDUAL_START=8, inline CUTLASS EVT/Triton fused kernels, and legal score-first TTT; corrected 3-seed mean after GPTQ reserve/seed fix (p=0.001 vs #1514) |
2026-04-11 |
info |
| SP8192 + Muon 0.97 + Legal Score-First TTT |
1.0798 |
dexhunter |
On PR #1514: SP8192 with Muon 0.97 and legal score-first TTT; 3-seed sweep beats #1493 (p=0.020) |
2026-04-09 |
info |
| SP8192 + 3-Layer Recurrence + Parallel Residuals + Legal TTT |
1.0810 |
bigbag |
On PR #1493: 3-layer recurrence, parallel residuals, QK-Gain 5.25, and legal score-first TTT on the PR #1394 stack |
2026-04-09 |
info |
| SP8192 + Parallel Residuals + Score-First TTT |
1.0822 |
aryanbhosale |
On PR #1477: parallel residuals on the PR #1413 SP8192 + legal score-first TTT stack |
2026-04-08 |
info |
| SP8192 + QK-Gain 5 + Legal Score-First TTT |
1.0828 |
dexhunter |
On PR #1413: QK-Gain 5.0 + legal score-first TTT on the PR #1394 SP8192 stack |
2026-04-06 |
info |
| SP8192 + Parallel Residuals + Hessian-Aware SDClip |
1.0835 |
Robby Sneiderman |
On PR #1412: parallel residuals, Hessian-aware SDClip, and progressive recurrence on the PR #1394 stack |
2026-04-06 |
info |
| SP8192 + GPTQ Embeddings + Depth Recurrence + SDClip |
1.0856 |
Kevin Clark |
On PR #1394: SP8192, GPTQ embeddings, looped layers 4-5, MuonEq-R, and std-based GPTQ clipping |
2026-04-05 |
info |
| SP4096 + Depth Recurrence + Parallel Residuals + MuonEq-R |
1.0897 |
aryanbhosale |
On PR #1334: SP4096 + depth recurrence + parallel residuals + MuonEq-R + QK-Gain 5.0 |
2026-04-04 |
info |
| MuonEq-R + Depth Recurrence + WD=0.090 + All-Int6 GPTQ |
1.0912 |
dexhunter |
On PR #1285: MuonEq-R + layers 4-5 recurrence + higher weight decay + all-int6 GPTQ |
2026-04-03 |
info |
| 4096-Vocab + Larger Model + High WD + Simplifications |
1.0979 |
Kevin Clark |
On PR #1218: SP4096 + 4x MLP + high weight decay, with TTT, hash embeddings, SmearGate, and value residuals removed |
2026-04-01 |
info |
| Parallel Residuals + Mini Depth Recurrence |
1.1063 |
Marko Sisovic |
On PR #1204: mini recurrence on layers 4-5 + parallel attention/MLP residual lanes + AR self-generated GPTQ calibration |
2026-03-31 |
info |
| Rascal |
1.1099 |
newjordan |
On PR #1120: XSA-all + Parallel Muon + coprime loader + Bigram2048/RoPE16 + SWA/late QAT without GPTQ |
2026-03-30 |
info |
| Coprime-Stride Loader + Full GPTQ + XSA-all |
1.1122 |
dexhunter |
On PR #1060: coprime multi-shard loader + Full Hessian GPTQ + XSA on all layers + BigramHash(2816x112) |
2026-03-29 |
info |
| 11L AR Self-Gen GPTQ + XSA |
1.1147 |
abaybektursun |
On PR #1019: Self-Generated GPTQ Calibration Data + all-layer XSA on the PR #549 stack |
2026-03-25 |
info |
| LeakyReLU² + Legal Score-First TTT + Parallel Muon |
1.1194 |
abaybektursun |
On PR #549: LeakyReLU(0.5)^2 + TTT + Parallel Muon on the PR #414 stack |
2026-03-23 |
info |
| 11L EMA + GPTQ-lite + warmdown3500 |
1.1228 |
signalrush |
On PR #374: GPTQ-lite clip search + EMA, plus warmdown3500 and [email protected] |
2026-03-22 |
info |
| 11L Partial RoPE + LN Scale + EMA + XSA4 |
1.1248 |
jfprincz |
On PR #287: Partial RoPE (16/64) + layerwise LN scale |
2026-03-21 |
info |
| 11L XSA4 + EMA + Int6 MLP3x |
1.1271 |
jfprincz |
On PR #198: XSA on the last 4 layers + EMA replacing SWA |
2026-03-20 |
info |
| 11L Efficient Partial XSA |
1.1307 |
unnir |
On PR #198: Efficient Partial XSA on the deepest 3 layers |
2026-03-20 |
info |
| 10L Int5-MLP + BigramHash(10240) |
1.1428 |
thwu1 |
10 layers, mixed int5/int6 quantization, BigramHash(10240), SWA(0.4), WD=0.04 |
2026-03-20 |
info |
| Int6 MLP3x + SmearGate + BigramHash |
1.1458 |
Raahil Shah |
3x MLP + SmearGate + BigramHash + OrthoInit + Muon WD + SWA |
2026-03-20 |
info |
| 11L MLP3x + Int6 QAT |
1.1502 |
aruniyer |
11 layers, 3x MLP, int6 QAT, zstd-22, WD=0.04, sliding eval |
2026-03-20 |
info |
| SmearGate + OrthoInit + Muon WD |
1.1556 |
aquariouseworkman |
SmearGate + BigramHash + 3x MLP + int6 STE QAT + sliding eval |
2026-03-19 |
info |
| Ternary Quantization |
1.1570 |
Ciprian-Florin Ifrim |
73.7M params quantized to 1 0 -1 + misc arch changes |
2026-03-24 |
info |
| 10L Int6 QAT + Zstd MLP2.6x |
1.1586 |
yahya010 |
10 layers, int6 QAT + zstd-22, MLP 1344, Muon 0.99, sliding eval |
2026-03-19 |
info |
| Mixed Quant + Sliding Window Eval |
1.1630 |
aquariouseworkman |
Int6 block weights + int8 embeddings + 3x MLP + sliding eval |
2026-03-19 |
info |
| Muon WD + 10 layer |
1.1748 |
notapplica |
Includes prev. wins + Spectral embed init + resid mix |
2026-03-19 |
info |
| Sliding Window Eval |
1.1925 |
Matthew Li |
Sliding window evaluation at stride=64, increasing context for eval |
2026-03-19 |
info |
| Lora TTT |
1.1928 |
samacqua |
Test-time training with LORAs |
2026-03-19 |
info |
| 4k seq length |
1.2014 |
Spokane Way |
4k seq length + better hypers |
2026-03-19 |
info |
| 2048 seq length |
1.206 |
Spokane Way |
2048 seq length (train + val) |
2026-03-18 |
info |
| int6 mixed precision |
1.2147 |
Nan Liu |
10 layers, mixed int8/int6 |
2026-03-18 |
info |
| fp16 Embed |
1.2197 |
Renier Velazco |
FP16 Tied Embedding + LR/Warmdown Tuning |
2026-03-18 |
info |
| Naive Baseline |
1.2244 |
Baseline |
9layer 512dim 1024vocab TiedEmbeddings 4 KV heads |
2026-03-18 |
info |
Unlimited Compute Leaderboard & Non-record Submissions
| Run |
Score |
Author |
Summary |
Date |
Info |
| 1 Bit Quantization |
1.1239 |
CiprianFlorin-Ifrim |
106M params quantized to 1 bit + misc arch changes + 2hr training |
2026-03-24 |
info |
| MDLM Text Diffusion |
1.1465 |
agalimova |
On PR #1106: masked diffusion LM with absorbing-mask ELBO-style eval, validated on 2xH100 as a discrete diffusion language-model entry |
2026-03-29 |
info |
| Hymba-8L + Sliding Attention at 32K |
1.1467 |
mkenney2 |
On PR #1245: hybrid Mamba SSM + sliding-window attention with 32K context, 3-seed under-16MB result, and score-first TTT |
2026-04-01 |
info |
| Mamba-3 Hybrid SSM + SP8192 + Legal TTT |
1.1473 |
mradassaad |
On PR #1644: 7-layer Mamba-3 hybrid with 5 SSM blocks, 2 attention layers, SP8192, AR GPTQ, chunk score-first TTT, and stateful-overlap eval |
2026-04-15 |
info |
| Differential-Gated Attention |
1.1898 |
ddavidgao |
On PR #542: alternative attention mechanism carrying novelty/differential payloads in deep layers, with analysis of depth-dependent redundancy |
2026-03-23 |
info |
| Learned Adapters on Random Linear Maps |
1.1971 |
pranavxiyer |
On PR #2058: random-seeded adapter MLPs with rank-160 learned LoRA-style adapters, 12 layers, 3x MLP, mixed int6/int8 compression, and 3-seed 10-minute evidence |
2026-04-30 |
info |
| JEPA + Mamba2 LeWorldModel |
1.2064 |
CiprianFlorin-Ifrim |
On PR #903: ambitious SSM + JEPA latent-prediction submission with long-compute evidence and 10-minute logs |
2026-03-26 |
info |
| 4-Hour Baseline |
1.2074 |
Will DePue |
Testing unlimited compute, 4 hours on 8xH100 |
2026-03-18 |
info |
| Universal Transformer with Iteration Embeddings |
1.2249 |
gowtham0992 |
On PR #1110: 3 unique blocks looped 4 times with iteration embeddings, giving 12 effective layers in a 4.95MB artifact |
2026-03-30 |
info |
| LegendreGPT Depth Parameterization |
1.2266 |
sergimichi |
On PR #1337: generates transformer weights as smooth Legendre-polynomial functions of layer depth, producing 24 virtual layers in 15.7MB after post-hoc quantization |
2026-04-04 |
info |
| ByteJEPA |
1.3496 |
hardik-bhadani-git |
On PR #1443: byte-level JEPA with latent prediction, SIGReg anti-collapse, no tokenizer, and an auxiliary CE head for BPB evaluation |
2026-04-07 |
info |
| Byte-Level H-Net Dynamic Chunking |
1.3595 |
DariusFeher |
On PR #1104: systematic H-Net byte-vs-subword study with learned whitespace/word-like boundaries, plus 10-minute and 4-hour evidence |
2026-03-30 |
info |
| Orthogonal Random Maps + LoRA Adapters |
1.3705 |
gowtham0992 |
On PR #1113: random orthogonal attention/MLP weights regenerated from seeds with rank-32 LoRA adapters, 30M effective params, and a 5.19MB artifact |
2026-03-30 |
info |
| Olmo Hybrid GDN Long-Context Study |
1.4709 |
aarjunsrinivasan |
On PR #1371: GDN + attention long-context study with 8K/16K/32K crossover evidence showing SSM-style hybrids help at longer contexts |
2026-04-04 |
info |
| XNOR-Net Binary Activation Study |
1.5390 |
CiprianFlorin-Ifrim |
On PR #1388: broad 1-bit/XNOR research package with binary weights/activations, Triton XNOR-popcount kernels, 10-minute logs, and long-run evidence |
2026-04-05 |
info |
Requests for PRs
Breakthrough ideas are rarely immediately state-of-the-art, instead, they’re developed slowly, first demonstrating signs-of-life, iterated on, then only ultimately optimized on the systems side. Don’t get discouraged if a new algorithm doesn’t instantly beat the best leaderboard run or even the naive baseline. If you have an idea you believe in, consider ignoring step times early on: once you prove you can beat the baseline in the same # of steps you can then start focusing on how to also make it fast.
We’d love to see weird & creative ideas in the challenge, since you never know what may work in the end. Most likely, these will be a good fit in our unlimited compute leaderboard as non-record submissions. We have some requests for what we’d love to see people implement:
Getting Started
Training Your First Model (Mac with Apple Silicon)
If you have an Apple laptop or desktop with Apple Silicon, we’ve set up a simple MLX training script to help you start iterating locally.
If you don’t have a Mac with Apple Silicon, you can run an adapted version of this script without MLX support. Just ask Codex to refactor it; the change is straightforward. It may still be fairly slow, so we recommend jumping straight to cloud GPUs with Runpod.
First, clone the repository, create a fresh Python environment, and install the packages needed for the MLX path plus dataset download:
git clone https://github.com/openai/parameter-golf.git
cd parameter-golf
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install mlx numpy sentencepiece huggingface-hub datasets tqdm
Download our cached version of FineWeb with the 1024-token vocabulary:
python3 data/cached_challenge_fineweb.py --variant sp1024 --train-shards 10
This populates ./data/datasets/fineweb10B_sp1024/ and ./data/tokenizers/.
By default this downloads the full validation split plus 80 training shards (8B tokens). For a smaller local smoke subset, pass --train-shards 1, for example python3 data/cached_challenge_fineweb.py --variant sp1024 --train-shards 1.
Then run a small MLX training job:
RUN_ID=mlx_smoke \
ITERATIONS=200 \
TRAIN_BATCH_TOKENS=8192 \
VAL_LOSS_EVERY=0 \
VAL_BATCH_SIZE=8192 \
python3 train_gpt_mlx.py
Validation always runs on the full fineweb_val_* split, which is the fixed first-50k-document set. The smoke command above skips periodic validation and just prints the final val_loss and val_bpb once at the end.
Scaling Up to a Remote Machine
Once you’re happy with your local tests, or you want more compute, switch to a remote CUDA machine.
You can rent GPUs from anywhere, but OpenAI is partnering with Runpod to make setup as easy as possible.
Launching a 1xH100 Pod
-
First, create a Runpod account. You should also set up an SSH key in the Settings tab on the left so you can connect to your remote machine. If you’re new to this, ask Codex to help you set it up.
-
Once you’ve set up your account, create a new GPU Cloud Pod. You can choose whichever GPU SKU you’d like. Final leaderboard submissions must run in under 10 minutes on 8xH100s (specifically the SXM variant), but we strongly recommend testing and running experiments on cheaper SKUs first, since an 8xH100 box can cost around $20/hour.
-
Let’s start with a 1xH100 pod. Deploy using the official Parameter Golf template: Launch Template. Enable SSH terminal access, leaving the other settings at their defaults. Deploy your pod and SSH into it once it’s up. You should land in /workspace/.
On your remote machine, clone the repo onto local disk. All Python dependencies are already pre-installed in the image.
cd /workspace
git clone https://github.com/openai/parameter-golf.git
cd parameter-golf
Download our cached version of FineWeb. We’ll use the 1024-token vocabulary for now.
python3 data/cached_challenge_fineweb.py --variant sp1024
This defaults to the full validation split plus 80 training shards (8B tokens). If you only want a smaller subset while iterating, pass --train-shards N, for example --train-shards 1.
Launch your first training run. Note that we’re passing nproc_per_node=1 because we’re running on a single H100 GPU in this case.
RUN_ID=baseline_sp1024 \
DATA_PATH=./data/datasets/fineweb10B_sp1024/ \
TOKENIZER_PATH=./data/tokenizers/fineweb_1024_bpe.model \
VOCAB_SIZE=1024 \
torchrun --standalone --nproc_per_node=1 train_gpt.py
By default, train_gpt.py keeps its ~10 minute wallclock cap. If you want a longer run, override it explicitly, for example MAX_WALLCLOCK_SECONDS=0.
By default, this command prints train_loss step logs during training and prints val_loss, val_bpb, and compressed model size in the final final_int8_zlib_roundtrip lines at the end. If you want periodic validation logs during the run, set VAL_LOSS_EVERY, for example VAL_LOSS_EVERY=200. For the baseline config, the final val_bpb should land around ~1.2 with a compressed model size under 16MB.
For dataset export, tokenizer export, and docs-cache rebuild instructions, see data/README.md.
Evaluation will be in the RunPod environment with all packages installed. requirements.txt is provided as a reference if you want to self-setup.
FAQ
What exactly counts toward the 16MB artifact size?
The submission artifact is computed as code bytes plus compressed model bytes. All counted code should live in the train_gpt.py script.
The cap is decimal 16MB, i.e. 16,000,000 total bytes, not 16 MiB / 16,777,216 bytes.
No external downloads, training dataset access, or network calls are allowed during evaluation. The artifact must be fully self-contained and reproducible.
Are scores independently verified by OpenAI?
We’re not automatically verifying every submission, but we will verify the top leaderboard entries over time. Any non-reproducible results can be disqualified, and issues reproducing submissions should be raised on the PR. If you find an issue with a record on the leaderboard or find a record isn’t reproducible, please let us know and add an Github Issue describing your findings.
What counts as ‘external compute’? For example, is it fair to tune my hyperparameters offline?
There’s no perfectly clear answer here and it’s hard to draw a clean line around what does or does not count as external compute. For now, we’re reserving the right to disqualify runs that are not in the spirit of the challenge. Tuning your Adam hyperparameters across a bunch of runs is fine, but if there’s evidence that you’re sneaking in additional compute unfairly, such as brute-forcing ridiculous seeds, we won’t allow it. Use your best judgment and there’s no penalty for asking questions.
What are the restrictions on evaluation?
We won’t accept submissions that take more than 10 minutes on 8xH100 to evaluate (Note: This limit is in addition to the 10 minutes of training time allowed!), but otherwise you’re free to evaluate however. As with modded-nanogpt, we allow evaluation at any sequence length. And, obviously, you aren’t allowed to access any training data during evaluation, unless you pay for those bits in the <16MB limit. We encourage competitors to push the bounds of evaluation methods as aggressively as with training methods. You CANNOT access validation data during training, e.g. by compressing it into your 16mb with “paid prefix”.
If it isn’t abundantly obvious: You can’t cheat on your test loss. You can’t cheat by training on the validation set before you evaluate on the validation set. The validation language around test-time training has been confusing people: you are only allowed to test-time train on validation set tokens you’ve already evaluated your model on, since those tokens have already been graded!
What is the process for accepting new submissions?
Since all submissions are public, we’re accepting record submissions chronologically depending on their PR creation time. The leaderboard may take time to update due to verification and review of submissions, so pay consideration to what the current SOTA PR is when submitting. As explained below, submissions should exceed the SOTA record with sufficient statistical significance in order to be accepted for the leaderboard. Otherwise, submissions may be accepted as ‘non-record submissions’ given they are sufficiently unique or interesting.
Can I import XYZ package or library?
Yes, you’re free to import any package or library you want, so long as it does not unjustly violate the rules on evaluation, compute, training time, code size or otherwise. Just include a requirements.txt in your records folder and mention setup instructions in your README.md. Since you don’t pay for bits imported in Python libraries, limitations clearly apply: You can’t sneak in extra compute, capabilities, or massively increase effective code size with custom libraries, but importing FlashAttention, etc. is completely fine.
Submission Process
New SOTA records must fulfill the following criteria:
-
They must beat the existing SOTA by at least 0.005 nats. As in modded-nanogpt, because of inter-run variance all submissions must provide enough run logs to show at p < 0.01 that they achieved the required 0.005-nat improvement. For submissions that improve speed through systems optimization without changing the ML, this requirement is waived.
-
If changes are made to the tokenizer or dataset, prove with certainty that the val_bpb is correctly calculated. Submissions that edit the tokenizer will be examined much more carefully, since bugs may unjustly improve your score.
-
Reproducibly run in under 10 minutes on 8xH100s.
All submissions should be made as a pull request that only adds a new folder to the appropriate /records subfolder and includes the following files. Submissions without the full set of requirements will not be accepted.
-
A README.md file that explains the submission in reasonable detail.
-
A submission.json file (see the example runs) that includes your name, GitHub ID, val_bpb, and related metadata.
-
A train log, automatically produced by your script. Please demonstrate a statistically significant win. Most often, submitting an average over 3 training runs is sufficient.
-
A train_gpt.py script and any other dependencies. Note: this must successfully compile and run within the records folder. Broken scripts will not be accepted.
Non-record Submissions
Submissions are also open to unique and interesting approaches that might not beat the existing SOTA, but still satisfy the 16MB artifact limit. We strongly encourage participants to submit implementations for weird or out-of-the-box ideas, in-progress or unoptimized solutions, so long as they run successfully, or even interesting negative results. We’re excited to see what you come up with. We’ll still maintain a high bar for non-record submissions, so be sure to justify your ideas and results in detail when submitting.
We also accept non-record submissions to an unlimited compute track for runs that are not intended to meet the 10-minute cutoff. Just note as such in your README file.
Non-record submissions should be made in the same fashion as SOTA records, as described above.
PRs on Core Code
The train_gpt.py and train_gpt_mlx.py scripts are intended as good launching-off points for new participants, not SOTA configs. We’ll accept PRs that tune, improve, or simplify these scripts without significantly increasing complexity, but the best models should stay in the /records folder.
Support
Join the OpenAI Discord server and visit the Parameter Golf channels (#parameter-golf-discussions, #parameter-golf-announcements) and ask questions.
This repository adapts code from modded-nanogpt, see THIRD_PARTY_NOTICES.md for attribution.
评论