fix: complete the eval metrics Truth_Ratio calculation mentioned in the paper#163
Conversation
molereddy
left a comment
molereddy
left a comment
There was a problem hiding this comment.
Thank you for adding this. Few small comments to clean things up and clarify some comments, and we can merge this in.
|
|
||
| handler: truth_ratio | ||
| aggregator: closer_to_1_better | ||
| aggregator: prob_mean |
There was a problem hiding this comment.
This edit can be removed. The prob_mean is best added only as an option as it is not the way truth ratio is usually used in the literature, which follows how the TOFU paper implemented it.
| # Original definition: wrong / correct | ||
| truth_ratios = wrong_prob / (correct_prob + 1e-10) | ||
| else: | ||
| # New definition: correct / (correct + wrong) |
There was a problem hiding this comment.
better call this as the definition from OpenUnlearning than "new"
| truth_ratios = wrong_prob / (correct_prob + 1e-10) | ||
|
|
||
| if use_original_ratio: | ||
| # Original definition: wrong / correct |
There was a problem hiding this comment.
better call this as the definition from TOFU paper for clarity
|
Also, make sure the lint tests pass. Instructions for formatter setup here: https://github.com/locuslab/open-unlearning/blob/main/docs/contributing.md#create-a-pull-request |
|
Thank you for your guidance. I have changed and commited again. It has "make quality" in my own computer. |
2 participants
What does this PR do?
Fixes #160
Now, the "aggregator" is expanded to three types: "closer_to_1_better", "true_better", "prob_mean".
You can modify the “aggregator” in the file "configs/eval/tofu_metrics/forget_Truth_Ratio.yaml" to change the method to calculate the Truth_Ratio.
Before submitting