Skip to content

A judge that knows the section of the assumptions, and counts the references that stand in for what a passage says - #127

Open
PierreMardon wants to merge 1 commit into
mainfrom
fix/judge-follows-the-blueprint
Open

PierreMardon wants to merge 1 commit into
mainfrom
fix/judge-follows-the-blueprint

Conversation

@PierreMardon

Copy link
Copy Markdown
Contributor

#123 changed the blueprint the extractor draws, and the judge's method did not follow (#124).

  • The frame. count.md and the no-padding criterion described the frame as the idea, the criteria and the scope, then the body and the sensitive zones. The section "Assumptions that approval confirms" was not in it: its callout and its items, which name rules the page states elsewhere, could be counted as padding.
  • Pointers. count.md gave a short pointer such as "see criterion 3" as the way not to repeat. The assumptions approval confirms gathered after the opening, and a blueprint understood where each passage stands #123 asks for the opposite: a fact named in a few words where a passage needs it, and a reference that adds to a sentence, never stands in for it. Read as it was, the method counted the few words as a repeat and let the pointer through.
  • Nothing measured the defect. No measure of the count saw a reference that stands in for what a sentence says.

What changes

The method

  • The section of the assumptions is part of the frame, in count.md and in the rubric. Its callout, its items and the mark where a rule is stated say no rule twice and are not padding. What an item says past the name of its rule, what left it open and the alternative set aside, is a fact like any other.
  • A fact named in a few words where a passage needs it is no repeat, and no pointer is given as the way out of one.
  • no-invented-rule speaks of an assumption that approving the blueprint confirms, no longer of one "to confirm".

A fourth measure

  • references_standing_in: the passages that, their reference taken away, no longer say what they mean. Such a passage refers by the number of a criterion, a label, a code or a title, or sends the reader to a file they do not read.
  • A reference that adds to a sentence complete without it is not counted. The judge lists each one with its passage, and the script counts them as it counts the details.
  • assumptions is a place a statement stands in.

The check of the judge

  • Each pair names the measures of the count its spoiled page must raise. The padded page raises the three it raised, and need hold no more references.
  • A new page, blueprint-referring.md, holds the facts of the lean one, with references in place of the words that name them. It must raise the references. It spoils no criterion of the rubric, so the count alone checks it.
  • An original that two pairs share is counted once.
  • The lean and the padded pages carry their assumption in the form of today, so the originals of the check hold the section the method now sets aside.

The report

  • It folds the measures that a run counted before holds.
  • It names as not compared a measure of the count that one summary alone holds. Against every campaign kept, that is references_standing_in.

The pages

  • evals/README.md defines the measure and the check.
  • evals/reports/README.md says what every campaign kept cannot be compared on. A move of blueprint.no-padding, repeated_in_prose or skippable_percent may come from the judge.
  • ARCHITECTURE.md changes in two sentences: what the count counts, and what the check holds of it.

Checked

  • scripts/gate.sh, green locally: 1,902 tests on 3.11 and on the newest Python.

Not measured: no judge check and no campaign was run. Both are billed and wait for your go. The next step of #124 is a campaign that measures #123 against 2026-10-07-*.

Closes #124

…erences that stand in for what a passage says

#123 changed the blueprint the extractor draws, and the judge's method did not follow. It
described a frame without the section "Assumptions that approval confirms", whose callout and
items name rules the page states elsewhere and could be counted as padding. It gave a short
pointer such as "see criterion 3" as the way not to repeat, where #123 asks for a fact named in
a few words, and for a reference that adds to a sentence and never stands in for it. And no
measure saw a reference that stands in: a campaign would have pushed the extractor back.

- evals/judge/count.md and evals/judge/rubric.md describe the frame with the section of the
  assumptions. Its callout, its items and the mark where a rule is stated say no rule twice
  and are not padding. A fact named in a few words where a passage needs it is no repeat, and
  no pointer is given as the way out of one. The interview's no-invented-rule speaks of an
  assumption that approving the blueprint confirms.
- A fourth measure of the count, `references_standing_in`: the passages that, their reference
  taken away, no longer say what they mean. The judge lists each with its passage and the
  script counts them. `assumptions` is a place a statement stands in.
- The judge check: each pair names the measures of the count its spoiled page must raise. The
  padded page raises the three it raised. A new page, drawn from the lean one with its facts
  and references in place of the words that name them, raises the references, and the count
  alone checks it. An original two pairs share is counted once. The lean and the padded pages
  carry their assumption in the form of today.
- The report folds the measures a run counted before holds, and names as not compared a
  measure of the count that one summary alone holds.
- evals/README.md defines the measure and the check, evals/reports/README.md says what every
  campaign kept cannot be compared on, and ARCHITECTURE.md names the fourth measure and what
  the check holds of the count.

Closes #124
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The judge knows neither the section of the assumptions nor a reference that stands in for what a sentence says

1 participant