Claude/great allen 51nxz7#194
Open
tinchen777 wants to merge 16 commits into
Open
Conversation
- Add demos/ with finetune, eval, and LoRA examples plus a Chinese code walkthrough (config system + evaluation metrics). - Add optional, backward-compatible LoRA support to get_model via a peft_args block, plus a Llama-3.2-1B-Instruct-LoRA model config.
Add LoRA finetuning support and comprehensive usage demos
- demos/4_unlearn.sh: run an existing method (GradDiff). - demos/5_custom_unlearn.sh + BoundedGradDiff trainer (class, registration, config) as a worked example of adding a custom unlearning algorithm. - Expand demos/README.md: unlearning flow, how to add your own method and how it gets called, config structure walkthrough, and eval output notes.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR adds a custom unlearning trainer example (BoundedGradDiff), introduces optional LoRA/PEFT wrapping in model loading, and provides runnable demo scripts/docs to showcase training/eval/unlearning workflows.
Changes:
- Added
BoundedGradDifftrainer and registered it in the trainer registry, with a matching Hydra config. - Added optional PEFT/LoRA adapter wrapping in
get_model()plus a LoRA model config. - Added a demos walkthrough (README + shell scripts) and minor config/.gitignore updates.
Reviewed changes
Copilot reviewed 13 out of 15 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/trainer/unlearn/bounded_grad_diff.py | Introduces BoundedGradDiff custom unlearning loss with bounded forget term. |
| src/trainer/init.py | Registers the new trainer so it can be referenced via Hydra handler. |
| src/model/init.py | Adds optional PEFT/LoRA wrapping of loaded models via peft_args. |
| demos/README.md | Provides detailed repo walkthrough and demos for finetune/eval/unlearn/LoRA/custom method. |
| demos/1_finetune.sh | Adds finetune demo script (but currently runs an extra unlearn command at the end). |
| demos/2_eval.sh | Adds eval demo script. |
| demos/3_lora_finetune.sh | Adds LoRA finetune demo script aligned with new PEFT support. |
| demos/4_unlearn.sh | Adds GradDiff unlearning demo script. |
| demos/5_custom_unlearn.sh | Adds custom BoundedGradDiff unlearning demo script. |
| configs/unlearn.yaml | Fixes indentation for trainer.args. |
| configs/trainer/BoundedGradDiff.yaml | Adds Hydra trainer config for the new custom method. |
| configs/model/Llama-3.2-1B-Instruct.yaml | Changes default model/tokenizer IDs to unsloth/.... |
| configs/model/Llama-3.2-1B-Instruct-LoRA.yaml | Adds a LoRA-enabled model config with peft_args. |
| configs/experiment/examples/tofu_eval.yaml | Updates tokenizer model ID to unsloth/.... |
| .gitignore | Normalizes .vscode/, adds .history/ and *.whl. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+40
to
+43
| forget_outputs = model(**forget_inputs) | ||
| # 有界梯度上升: NLL 一旦超过 tau，clamp 后梯度为 0，避免无限制崩坏 | ||
| bounded_nll = torch.clamp(forget_outputs.loss, max=self.forget_loss_bound) | ||
| forget_loss = -bounded_nll |
Comment on lines
+35
to
+39
| forget_inputs = { | ||
| "input_ids": forget_inputs["input_ids"], | ||
| "attention_mask": forget_inputs["attention_mask"], | ||
| "labels": forget_inputs["labels"], | ||
| } |
Comment on lines
+47
to
+51
| retain_inputs = { | ||
| "input_ids": retain_inputs["input_ids"], | ||
| "attention_mask": retain_inputs["attention_mask"], | ||
| "labels": retain_inputs["labels"], | ||
| } |
Comment on lines
+94
to
+95
| with open_dict(peft_args): | ||
| adapter_path = peft_args.pop("path", None) |
| model = PeftModel.from_pretrained(model, adapter_path, is_trainable=True) | ||
| logger.info(f"Loaded existing LoRA adapter from {adapter_path}") | ||
| else: | ||
| lora_config = LoraConfig(**peft_args) |
Comment on lines
+40
to
+41
| python src/train.py --config-name=unlearn.yaml experiment=unlearn/tofu/default \ | ||
| forget_split=forget10 retain_split=retain90 trainer=GradAscent task_name=SAMPLE_UNLEARN No newline at end of file |
| @@ -1,9 +1,9 @@ | |||
| model_args: | |||
| pretrained_model_name_or_path: "meta-llama/Llama-3.2-1B-Instruct" | |||
| pretrained_model_name_or_path: "unsloth/Llama-3.2-1B-Instruct" | |||
| torch_dtype: bfloat16 | ||
| tokenizer_args: | ||
| pretrained_model_name_or_path: "meta-llama/Llama-3.2-1B-Instruct" | ||
| pretrained_model_name_or_path: "unsloth/Llama-3.2-1B-Instruct" |
- evals/metrics/utils.py: cast losses/probs to float() before .numpy(), numpy has no bfloat16 so bf16 models crashed evaluate_probability (TypeError: Got unsupported ScalarType BFloat16). - model/__init__.py: stop passing cache_dir=$HF_HOME to from_pretrained so transformers uses the conventional $HF_HOME/hub location, matching where datasets are cached.
Covers prerequisites (setup_data), device selection (CPU/GPU, flash-attn caveats), HF_HOME cache layout, the two fixed bf16-related bugs, and task_name notes.
Plain 'python src/train.py' lets HF Trainer use nn.DataParallel across all visible GPUs; on a busy node that OOMs when replicating to an occupied card. Scripts now default CUDA_VISIBLE_DEVICES to a single card (overridable), and the README documents the failure mode.
transformers flipped apply_chat_template's return_dict default to True, so tokenize=True now yields a BatchEncoding instead of list[int], breaking 'chat_ids += [eos]' with TypeError. Pass return_dict=False explicitly (backward compatible; old default was already False).
- from_pretrained: pass dtype= instead of deprecated torch_dtype= (model loading and MIA reference model). Requires transformers>=4.56. - Remove deprecated TrainingArguments.logging_dir (removed in v5.2); TensorBoard now defaults under output_dir/runs. Note: the YAML 'torch_dtype' key is the project's own config consumed by get_dtype() and is intentionally left unchanged.
Hydra owns logging via logging.config.dictConfig (job_logging group), which overrode any user-set RichHandler. Add configs/hydra/job_logging/rich.yaml (rich.logging.RichHandler + file handler) and select it in hydra/default.yaml. Requires 'pip install rich'; revert per-run with hydra/job_logging=colorlog.
Add src/log_utils.py:RichNameFormatter which wraps the logger name in a Rich style tag (default 'bold cyan') and escapes the message so brackets in log text aren't parsed as markup. rich.yaml uses it for the console handler (markup=true) and a separate plain formatter for the file handler.
The rich option only existed in the job_logging group, so selecting hydra_logging=rich failed with 'Could not find hydra/hydra_logging/rich'. Add configs/hydra/hydra_logging/rich.yaml and set both groups to rich in hydra/default.yaml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes # (issue)
Before submitting