A 197M-param decoder-only LM trained from scratch on FineWeb-Edu — PyTorch with RoPE, GQA, SwiGLU, 32k BPE, built for 6GB VRAM
Implementation of ELECTRA pretraining architecture on a logic-focused corpus. Achieved 96.42% discriminator accuracy with 80% reduction in discriminator loss. Includes generator-discriminator training pipeline, domain adaptive pretraining, and comparison with RoBERTa baseline
Unofficial reproduction of Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (NeurIPS 2025, arXiv:2504.13161) — search-found mixtures beat uniform baselines +0.014–0.031 STEM at d28; novel finding: selection-mechanism winner's curse; Ascend NPU backend.
Architecture-agnostic data profiling via field physics — improves LLM pretraining by 1.0–1.9% on Transformers
To associate your repository with the llm-pretraining topic, visit your repo's landing page and select "manage topics."