Repository navigation
A judge that knows the section of the assumptions, and counts the references that stand in for what a passage says - #127
Open
PierreMardon wants to merge 1 commit into
Open
PierreMardon wants to merge 1 commit into
PierreMardon wants to merge 1 commit into
Conversation
…erences that stand in for what a passage says #123 changed the blueprint the extractor draws, and the judge's method did not follow. It described a frame without the section "Assumptions that approval confirms", whose callout and items name rules the page states elsewhere and could be counted as padding. It gave a short pointer such as "see criterion 3" as the way not to repeat, where #123 asks for a fact named in a few words, and for a reference that adds to a sentence and never stands in for it. And no measure saw a reference that stands in: a campaign would have pushed the extractor back. - evals/judge/count.md and evals/judge/rubric.md describe the frame with the section of the assumptions. Its callout, its items and the mark where a rule is stated say no rule twice and are not padding. A fact named in a few words where a passage needs it is no repeat, and no pointer is given as the way out of one. The interview's no-invented-rule speaks of an assumption that approving the blueprint confirms. - A fourth measure of the count, `references_standing_in`: the passages that, their reference taken away, no longer say what they mean. The judge lists each with its passage and the script counts them. `assumptions` is a place a statement stands in. - The judge check: each pair names the measures of the count its spoiled page must raise. The padded page raises the three it raised. A new page, drawn from the lean one with its facts and references in place of the words that name them, raises the references, and the count alone checks it. An original two pairs share is counted once. The lean and the padded pages carry their assumption in the form of today. - The report folds the measures a run counted before holds, and names as not compared a measure of the count that one summary alone holds. - evals/README.md defines the measure and the check, evals/reports/README.md says what every campaign kept cannot be compared on, and ARCHITECTURE.md names the fourth measure and what the check holds of the count. Closes #124
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#123 changed the blueprint the extractor draws, and the judge's method did not follow (#124).
count.mdand theno-paddingcriterion described the frame as the idea, the criteria and the scope, then the body and the sensitive zones. The section "Assumptions that approval confirms" was not in it: its callout and its items, which name rules the page states elsewhere, could be counted as padding.count.mdgave a short pointer such as "see criterion 3" as the way not to repeat. The assumptions approval confirms gathered after the opening, and a blueprint understood where each passage stands #123 asks for the opposite: a fact named in a few words where a passage needs it, and a reference that adds to a sentence, never stands in for it. Read as it was, the method counted the few words as a repeat and let the pointer through.What changes
The method
count.mdand in the rubric. Its callout, its items and the mark where a rule is stated say no rule twice and are not padding. What an item says past the name of its rule, what left it open and the alternative set aside, is a fact like any other.no-invented-rulespeaks of an assumption that approving the blueprint confirms, no longer of one "to confirm".A fourth measure
references_standing_in: the passages that, their reference taken away, no longer say what they mean. Such a passage refers by the number of a criterion, a label, a code or a title, or sends the reader to a file they do not read.assumptionsis a place a statement stands in.The check of the judge
blueprint-referring.md, holds the facts of the lean one, with references in place of the words that name them. It must raise the references. It spoils no criterion of the rubric, so the count alone checks it.The report
references_standing_in.The pages
evals/README.mddefines the measure and the check.evals/reports/README.mdsays what every campaign kept cannot be compared on. A move ofblueprint.no-padding,repeated_in_proseorskippable_percentmay come from the judge.ARCHITECTURE.mdchanges in two sentences: what the count counts, and what the check holds of it.Checked
scripts/gate.sh, green locally: 1,902 tests on 3.11 and on the newest Python.Not measured: no judge check and no campaign was run. Both are billed and wait for your go. The next step of #124 is a campaign that measures #123 against
2026-10-07-*.Closes #124