v0.2.0

v0.2.0 Release for RAID

This release adds:

• Support for multiple FPR values & AUROC in run_evaluation
• Leaderboard toggle to select one of three metrics: TPR@FPR=5%, TPR@FPR=1%, and AUROC
• Recalculation of all existing detector evaluation scores to include these new metrics
• Logic to remove detector predictions that do not meet a particular FPR threshold
• Warnings on submission if your detector fails to meet the FPR threshold

Non-breaking interface changes

• evaluate_cli.py now takes in multiple arguments for target_fpr.
• The default value for evaluate_cli.py is both 0.05 FPR and 0.01 FPR.

Potentially breaking changes

The output format of results.json now has a slightly altered structure. Instead of the accuracy field pointing to the TPR@FPR=5% it now points to a dictionary indexed by the FPR containing the true positives, false negatives, and TPR for the particular fpr value.

When detectors are unable to achieve a given target FPR, the resulting field in the accuracy dictionary will be given a null value (as seen below).

Old

{
  "domain": "abstracts",
  "model": "llama-chat",
  "decoding": "greedy",
  "repetition_penalty": "no",
  "attack": "none",
  "tp": 200,
  "fn": 0,
  "accuracy": 1.0
},

New

{
  "domain": "abstracts",
  "model": "llama-chat",
  "decoding": "greedy",
  "repetition_penalty": "no",
  "attack": "none",
  "accuracy": {
    "0.05": {
      "tp": 200,
      "fn": 0,
      "accuracy": 1.0
    },
    "0.01": null
  },
  "auroc": 0.9989833333333333
},

This is a BREAKING CHANGE for any code built off evaluate_cli or run_evaluation that directly accesses the results.json. Please take care to catch these null values and index accuracy correctly.