LAD: LoRA-Adapted Diffusion

Efficiently transforming autoregressive to diffusion models using only LoRA

This repository contains a modular implementation of a diffusion-style language model based on LLaMA, fine-tuned using instruction-following datasets performed by training only with LoRA adapters.

Demo

Check out the demo first: https://huggingface.co/spaces/Ruurd/tini-lad

Or the extensive demo at: https://huggingface.co/spaces/Ruurd/lad

🔧 Features

• LoRA-based lightweight fine-tuning
• Diffusion-style corruption and denoising of assistant responses
• Token-level structural noise (masking, swapping, duplication, span shift)
• Instruction-tuning with various datasets, such as Alpaca, Alpaca-GPT4, and Clean-Instruct datasets

🗂️ Project Structure

diffusion-llama 
├── configs
| └── model_config.py
├── models
| └── custom_transformer.py
├── data
│ ├── datasets.py
│ └── noise.py
├── training
│ ├── train.py
│ ├── collator.py
│ └── callbacks.py
├── notebooks
| └── demo_instruction_tuned.ipynb
| └── demo_unconditional.ipynb
├── utils
| └── tokens.py
├── requirements.txt
└── README.md
└── model_card.md

🚀 Quickstart

1. Install dependencies pip install -r requirements.txt

2. Run training python training/train.py

3. Run inference Try the Jupyter Notebooks: notebooks/demo_instruction_tuned.ipynb notebooks/demo_unconditional.ipynb

📦 Datasets

This project uses data from:

tatsu-lab/alpaca
vicgalle/alpaca-gpt4
crumb/Clean-Instruct-3M (streamed + filtered) and more...

🧪 Sample Output

Question : User: What is the capital of France?
Input : Assistant: MASK MASK MASK MASK MASK MASK
Iteration 1 : Assistant: The MASK of Lyon MASK is.
Iteration 2 : Assistant: MASK capital of France MASK MASK.
Iteration 3 : Assistant: The capital of MASK is Paris.
Final : Assistant: The capital of France is Paris.

📜 License

MIT License. Feel free to use, modify, and cite.