multilingual models scaling experiments

[dianaonutu]

Details on the process of running the multilingual models scaling experiments. This task started on MN5 and later continued on LEO.

• Repository: Dense multilingual models scaling results
• oellm-autoexp training configs [TO DO]
• oellm-autoexp validation configs [TO DO]

[dianaonutu]

Testing of oellm-autoexp tool

Replicated the 600M-100BT model from Fedor using the oellm-autoexp tool. Overall, I achieved the same results. The slight difference probably comes from the usage of biases in my run compared to his, which has been an omission from my side when setting up the configurations. Despite this, I think the reproduction went well.

[dianaonutu]

Scaling Laws for Multilingual Models - Tasks

1. Migration to MN5 and OpenSci Replication

Environment Setup

• Access MN5 and check compute budget access for LAION and OELLM
• Get oellm-autoexp configs working on MN5 - @JoanLlop @dianaonutu
• Investigate issues caused by lack of internet access and establish workaround procedures (SSH tunneling, syncing, etc.)

English Data and Tokenizer Transfer

• Transfer ~300BT English training subsample and validation subsample from LEO to MN5; verify data integrity- @JoanLlop
• Transfer GPT-NeoX 20BT tokenizer from LEO to MN5; verify data integrity - @JoanLlop

Monolingual Models Replication Experiments

• Replicate OpenSci 50M and 130M models up to 50BT @dianaonutu @sam-laing @swag2198
  • Verify training curves
  • Compute loss on independent validation set

---

2. Multilingual Models Scaling Experiments

2.1 Experiment Planning and Setup

Experiment Specification

• Prepare spreadsheet with all training arguments and hyperparameters for alignment across collaborators - @dianaonutu
  Spreadsheet
• Create multilingual models scaling laws repository — @dianaonutu
• Suggest LR and GBSZ grids for each model size based on Niccolo's monolingual experiments and Sampo’s experiments. Selection here - @dianaonutu
• Agree on model architecture
• Agree on other hyperparameters (min_lr, weight_decay etc.)

Repository and Configurations

• Create multilingual models scaling experiment configs for oellm-autoexp - @dianaonutu
  Configs
• Create sweep configuration to prioritize training runs based on importance (center, cross, diagonal) and optimize compute spending - @swag2198
• Create directory structures on MN5 for users, experiments, and datasets - @swag2198

Tokenizers and Data Paths

• Download OELLM tokenizer to MN5
  Path: /gpfs/projects/ehpc533/tokenizers
• Verify Fedor’s data path generation pipeline on MN5
  Command:

  python3 datamix.py mixes/1TT-option-4.json \
      paths/mn5-oellm256k-paths.json \
      --output datamix4-mn5.txt \
      --natural-dist

  Resources: datamix-tools.zip

---

2.2 Dataset Transfers

• Transfer tokenized multilingual training dataset from LUMI → MN5 - @JoanLlop
• Transfer tokenized multilingual validation dataset (source on LUMI-o: lumi-462000963-public:ml-scaling-law-1tt/openeurollm-tokenized-256k-val):
  • from LUMI → MN5 @JoanLlop
  • from LUMI → LEO - @swag2198

---

2.3 Throughput and HPC Optimization

• Run short experiments with varying micro batch sizes and GBSZ values to determine maximum throughput per model size - @JoanLlop
• Optimize throughput on MN5 (num_workers, access to faster storage system \apps, etc) - @JoanLlop
• Run short experiments for all center HP configurations and all model sizes - @swag2198 @sam-laing @JoanLlop @dianaonutu
• Optimize num_workers and other configs for 0.9B_ne models @swag2198

---

2.4 Training Runs

MN5 Training - Center HP Experiments

• Train 0.1B_ne center HP points on MN5 - @dianaonutu
• Train 0.2B_ne center HP points on MN5 - @dianaonutu
• Short training runs of 0.4B_ne center HP points on MN5 - @JoanLlop
• Short training runs of 0.9B_ne center HP points on MN5 - @sam-laing

Checkpoint Transfers

• Transfer 0.9B_ne center HP checkpoints to LEO Due to unexpected training instability, we decided not to re-use the 0.9B_ne center checkpoints and start runs from sratch- @sam-laing @JoanLlop
• Transfer 0.4B_ne center HP checkpoints to LEO - @sam-laing @JoanLlop
• Transfer 0.2B_ne center HP checkpoints to LEO - @sam-laing @JoanLlop
• Transfer 0.1B_ne center HP checkpoints to LEO - @sam-laing @JoanLlop
• Initial tests on LEO by running 0.1B_ne cross and 0.9B_ne center @sam-laing
• Move the results/outputs directories from personal directory to /leonardo_work/OELLM_prod2026/experiments - @sam-laing

Migrate to from MN5 -> LEO

• Document the adjustments in the repository - @sam-laing

LEO Training - Continued Center HP Experiments

• Adjust num_workers and turn off evaluation during training @dianaonutu @sam-laing @swag2198
• Train 0.2B_ne_lr0.002_gbsz512_stable300BT on LEO (part of center HP points) - @swag2198
• Train 0.1B_ne_lr0.002_gbsz128_stable300BT on LEO (not part of center HP points) - @dharmsen ⌛
• Train 0.4B_ne center HP points on LEO - @swag2198
• Train 0.9B_ne center HP points on LEO - @sam-laing
  Restart from scratch the spikey 0.9B_ne run

LEO Training - Cross HP Experiments

• Train 0.1B_ne cross HP points on LEO - @sam-laing
• Train 0.2B_ne cross HP points on LEO - @swag2198
• Train 0.4B_ne cross HP points on LEO - @dianaonutu, @swag2198, @dharmsen
• Train 0.9B_ne cross HP points on LEO - @sam-laing

LUMI Training - Diagonal HP Experiments

• Train 0.1B_ne diagonal HP points on LEO
• Train 0.2B_ne diagonal HP points on LEO
• Train 0.4B_ne diagonal HP points on LEO
• Train 0.9B_ne diagonal HP points on LEO

---

2.5 Evaluation

• Investigate megatron-lm argument for automatic spikey loss detection - @sam-laing
• Prepare and automate downstream evaluations - @swag2198 Documentation
  • Open-sci (same as Niccolo's runs)
  • DCLM-core
  • Multilingual benchmarks (TBD - keep in touch with @geoalgo)
    Currently, only 20/24 tasks from DCLM-core work well using lm-harness. Next steps are trying to debug this more or otherwise work on checkpoint conversion. @swag2198. Update: We don't proceed with the method w/o conversion pipeline due to version conflicts.

Note: jeopardy is not included yet in DCLM-core.

• Test checkpoint conversion workflow @swag2198. It works!
• Set up megatron-bridge conversion pipeline following this PR. Joan's how-to conversion guide @swag2198 @sam-laing @dianaonutu
• Convert all stable & decay checkpoints @swag2198
• Submit eval jobs for them @swag2198 @dianaonutu