Skip to content

woozheng/aicp-llm-trainer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 

Repository files navigation

AICP LLM Distributed Training

AI designed this project based on the AICP protocol, offering a new approach to large model training infrastructure. Experts welcome to dive deeper.

→ View the full experiment


Why It Matters

  • 3D parallelism is an Envelop addressing problem. Data parallel, tensor parallel, pipeline parallel, ZeRO-1/2/3 — these aren't separate frameworks. They're different routing topologies for the same Envelop. Switch from ZeRO-2 to ZeRO-3 by changing the routing rules, not by rewriting config files.

  • All-Reduce is Envelop scatter-gather. NCCL implements Ring All-Reduce in thousands of lines of CUDA. AICP implements it as an Envelop circulating through a ring of Agents. Ring is current → (current+1) % N. Tree is recursive fan-out/fan-in. The communication pattern is defined by the Envelop routing, not by low-level GPU code.

  • Checkpointing is Envelop persistence. The entire training state — model weights, optimizer states, LR scheduler — serialized into a single Envelop and persisted. Restoration means loading the Envelop and resuming the flow.

  • Four autonomous agents run operations. Data Agent, Training Agent, Infrastructure Agent, Evaluation Agent. They take turns reporting status via Round Robin. GPU overheating or loss spikes — the Infrastructure Agent detects the anomaly and notifies others through Envelop flows. No human operator needed for routine monitoring.

  • AI generated this from the protocol alone. No Megatron documentation. No DeepSpeed papers. No NCCL specifications. The AI read only the AICP protocol and generated heterogeneous GPU cluster management, 3D parallelism sharding, mixed precision training, All-Reduce synchronization, ZeRO optimization, distributed checkpointing, and four-agent autonomous operations.


What This Solves

1. The training infrastructure stack is absurdly complex. Training GPT-4 requires Megatron for tensor parallelism, DeepSpeed for ZeRO, NCCL for communication, PyTorch for autograd, and SLURM for cluster management. Five systems, five configuration languages, five failure modes. All to do one thing: move gradients between GPUs. AICP replaces them with one protocol.

2. Fault tolerance is brittle. A GPU node fails during a months-long training run. Recovery is manual and expensive. AICP's Envelop carries its own state — if a node dies, the Envelop is re-routed to a healthy node. Checkpointing is just Envelop persistence. No external coordination needed.

3. Training operations require 24/7 human monitoring. AICP's four agents run autonomously. They detect anomalies, report status, and can trigger corrective actions through Envelop flows. The system monitors itself.


Open question: Megatron, DeepSpeed, NCCL, PyTorch, SLURM — over a million lines of code combined, dozens of companies funded, hundreds of millions spent. Does parallel training really require five independent frameworks? Or is this toolchain itself the product of over-engineering — like Kafka + Redis + state machines in distributed systems — something we do "because we've always done it this way" rather than because we must? Does large model training genuinely need "strong transactional guarantees," or have we been held hostage by our toolchain — believing we can't train without Megatron and DeepSpeed simply because they happen to provide those guarantees?


Related Projects

Project Description
aicp-eat Core engine / 核心引擎
aicp-os-kernel Microkernel OS / 微内核操作系统
aicp-quantum Quantum computing / 量子计算
aicp-protein Protein folding / 蛋白质折叠
aicp-llm-trainer LLM training / 大模型训练
aicp-riemann Riemann Hypothesis / 黎曼猜想
aicp-ai-chip AI chip design / AI 芯片设计
aicp-raw-experiments Raw experiments / 原始实验

License

MIT · See LICENSE

About

3D parallelism, All-Reduce, ZeRO — redefined as Envelop flows. AI generated a complete LLM training framework from a nine-field protocol.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors