Repository navigation
Conversation
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit eb92b0c. Configure here.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Hi! Thanks for porting and maintaining
qed_math_envand its test suite.While reading
MathProofRubric(envs/qed_math_env/server/rubric.py), I noticed a small edge case in how<score>N</score>tags are extracted:MathProofRubric._parse_responseusesre.search(r"<score>(\d+)</score>", text), so the first<score>N</score>anywhere in the grader response becomes the grade._build_promptembeds the submitted proof verbatim. Underv0.md(where<assessment>precedes<score>) or the fallback prompt, if the grader quotes a<score>7</score>from the proof inside its assessment before emitting its verdict<score>1</score>,re.searchpicks the quoted7.v1.md(the default) andv2.md,<score>comes first before<assessment>, so switching blindly to last-match would breakv1.md/v2.mdif the grader quotes a tag inside<assessment>.<assessment>...</assessment><score>6</score>(v0/ fallback)<assessment>Proof writes <score>7</score>...</assessment><score>1</score><score>1</score><assessment>Proof writes <score>7</score>...</assessment>(v1/v2)This PR:
<assessment>...</assessment>and<errors>...</errors>blocks in_parse_responsebefore extracting<score>N</score>(taking the last remaining match for unstructured fallback responses).<score>,<assessment>, and<errors>tags (case-insensitive) fromproofin_build_prompt.Scope note: this prevents quoted or injected XML tags in the proof from interfering with score extraction across
v0.md,v1.md,v2.md, and fallback prompts; general prompt injection (where an LLM grader follows plain-text instructions in the proof to emit a higher score) remains a prompt-level limitation.Type of Change
RFC Status
Test Plan
Added unit tests in
tests/envs/test_qed_math_environment.py:v0shape: quoted<score>7</score>inside<assessment>before closing<score>1</score>->1v1/v2shape: leading<score>1</score>followed by quoted<score>7</score>inside<assessment>(including truncated<assessment>) ->1<score>7</score>followed by closing<score>1</score>->1[0, MAX_SCORE]) and missing-tag (0) behavior unchanged_build_promptstrips<score>,<assessment>, and<errors>tags (case-insensitive) fromproofNote
Medium Risk
Changes core reward parsing for proof grading; incorrect regex could mis-score submissions, but behavior is covered by new unit tests and existing clamping/missing-tag paths are unchanged.
Overview
Fixes incorrect proof scores when the LLM grader quotes
<score>tags from the submission inside<assessment>/<errors>prose, which previously made_parse_responsepick the first tag anywhere in the response instead of the verdict.MathProofRubricnow strips closed and unclosed assessment/errors blocks before finding<score>N</score>, uses the last remaining match (with a fallback path), and removes<score>,<assessment>, and<errors>tags from submitted proofs in_build_promptso injected markup cannot pollute the grader prompt.Unit tests cover v0/v1 response shapes, truncated assessments, self-closing tags, and prompt sanitization.
Reviewed by Cursor Bugbot for commit b04549e. Bugbot is set up for automated code reviews on this repo. Configure here.