Repository navigation
Conversation
…rld, connect4 and chess
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a34a9c6. Configure here.
5 of 12 tasks
A custom FEN can start the game in checkmate or stalemate, so reset() already returns done=True. Store that observation like a terminal step does, so step() returns it again instead of scoring a move.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
maze_env,grid_world_env,connect4_envandchess_envnever check instep()whether the episode has already ended, so a finished episode keeps playing, and in two of them keeps scoring:maze_env: after a win, stepping off the exit and back on pays+10.0again withdone=True. A lost episode keeps moving the agent.grid_world_env: after reaching the goal,UPthenDOWNpays+1.0again withdone=True.connect4_env: after a win or an invalid move, the nextstep()drops another piece and returnsdone=False, which reopens the game.chess_env: after checkmate, the nextstep()returnsdone=False,reward=-0.1andresult=None.Each
step()now keeps the terminal observation and returns a copy of it until the nextreset(). This is the same pattern #1552 used forreasoning_gym_env,finqa_envandqed_math_env, and the one #1685 proposes forwildfire_env.step_countstops at the final step. Each README and docs page now says what happens on further steps.Not covered here:
coding_tools_envreruns its verify commands on a secondsubmit_solution. I found this by reading the code and haven't reproduced it. It is an MCP environment, so its fix also depends on how direct/mcpcalls are handled (#1507).snake_envI couldn't check locally.Type of Change
Alignment Checklist
Before submitting, verify:
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles.claude/docs/INVARIANTS.mdand no invariants are violated/pre-submit-pr(orbash .claude/hooks/lint.shand tests) and addressed all issuesRFC Status
Test Plan
step()andreset():tests/envs/test_maze_step.py: the exit pays once, returned observations are copies, a lost episode stays lost, andreset()starts a fresh episode.TestStepAfterDoneintests/envs/test_grid_world.py: the goal pays once, andreset()starts a fresh episode.TestStepAfterDoneintests/envs/test_connect4_environment.py: the game stays finished after a win and after an invalid move.test_steps_after_checkmate_return_the_final_observationintests/envs/test_chess_environment.py.main, the 7 new tests fail; with this branch, all pass.PYTHONPATH=src:envs python -m pyteston the maze, grid_world, connect4 and chess tests plustests/envs/test_client_step_result_contract.pyandtests/envs/test_manifest_app_targets.py: 153 passed (Linux, Python 3.12).tests/minus the four ignored files,-m "not integration and not network and not docker"), run in parallel: 6383 passed, 122 skipped, 0 failed. The server-basedtest_maze_environment.py(needsgunicorn) and the legacytest_connect4_env.pyare among the skipped tests.bash .claude/hooks/lint.sh,ruff check,ruff format --checkon the changed files, andpython scripts/sync_env_docs.py --checkpass.usort checkflagstests/envs/test_grid_world.py, which it already flags onmain(listed inAGENTS.md).Claude Code Review
Alignment review by Opus 5.5, in the format of
.claude/skills/alignment-review/SKILL.md.Automated Checks
bash .claude/hooks/lint.shprint,breakpointor TODO lines.Open RFCs Context
No open RFC covers per-environment episode termination. RFC 014 and #1507 cover episode limits for direct MCP calls, which these four non-MCP environments don't have.
Tier 1: Fixes Required
None.
Tier 2: Alignment Discussion
Principle Conflicts
None identified. The
reset/step/statesignatures are unchanged, rewards stay inside each environment, andreset()still starts a clean episode.RFC Conflicts
None identified.
Summary
Note
Medium Risk
Changes core
step()semantics for four RL environments, which can affect training loops that step pastdone; behavior is intentional and aligned with other OpenEnv envs but is a correctness fix with broad test coverage.Overview
Chess, Connect4, Grid World, and Maze now treat a finished episode as closed until
reset(): each environment keeps a terminal_final_observationandstep()returns a deep copy of it withdone=True, the same reward/result, and no further moves or scoring (including duplicate goal/exit payouts and Connect4/Chess incorrectly continuing play).Documentation in the env READMEs and
docs/source/environments/describes this post-terminalstep()behavior.New tests cover steps after win/loss/checkmate/stalemate, invalid Connect4 moves, and observation copying in maze.
Reviewed by Cursor Bugbot for commit 71b22a5. Bugbot is set up for automated code reviews on this repo. Configure here.