Skip to content

fix(envs): stop scoring steps after the episode ends in maze, grid_world, connect4 and chess - #1693

Open
Aafimalek wants to merge 2 commits into
openenv:mainfrom
Aafimalek:fix/games-step-after-done
Open

Aafimalek wants to merge 2 commits into
openenv:mainfrom
Aafimalek:fix/games-step-after-done

Conversation

@Aafimalek

@Aafimalek Aafimalek commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

maze_env, grid_world_env, connect4_env and chess_env never check in step() whether the episode has already ended, so a finished episode keeps playing, and in two of them keeps scoring:

  • maze_env: after a win, stepping off the exit and back on pays +10.0 again with done=True. A lost episode keeps moving the agent.
  • grid_world_env: after reaching the goal, UP then DOWN pays +1.0 again with done=True.
  • connect4_env: after a win or an invalid move, the next step() drops another piece and returns done=False, which reopens the game.
  • chess_env: after checkmate, the next step() returns done=False, reward=-0.1 and result=None.

Each step() now keeps the terminal observation and returns a copy of it until the next reset(). This is the same pattern #1552 used for reasoning_gym_env, finqa_env and qed_math_env, and the one #1685 proposes for wildfire_env. step_count stops at the final step. Each README and docs page now says what happens on further steps.

Not covered here: coding_tools_env reruns its verify commands on a second submit_solution. I found this by reading the code and haven't reproduced it. It is an MCP environment, so its fix also depends on how direct /mcp calls are handled (#1507). snake_env I couldn't check locally.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • New environment
  • Refactoring

Alignment Checklist

Before submitting, verify:

  • I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
  • I have checked .claude/docs/INVARIANTS.md and no invariants are violated
  • I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

RFC Status

  • Not required (bug fix, docs, minor refactoring)
  • RFC exists: #___
  • RFC needed (will create before merge)

Test Plan

  • New tests, all through the environments' own step() and reset():
    • tests/envs/test_maze_step.py: the exit pays once, returned observations are copies, a lost episode stays lost, and reset() starts a fresh episode.
    • TestStepAfterDone in tests/envs/test_grid_world.py: the goal pays once, and reset() starts a fresh episode.
    • TestStepAfterDone in tests/envs/test_connect4_environment.py: the game stays finished after a win and after an invalid move.
    • test_steps_after_checkmate_return_the_final_observation in tests/envs/test_chess_environment.py.
  • With the four environment files taken from main, the 7 new tests fail; with this branch, all pass.
  • PYTHONPATH=src:envs python -m pytest on the maze, grid_world, connect4 and chess tests plus tests/envs/test_client_step_result_contract.py and tests/envs/test_manifest_app_targets.py: 153 passed (Linux, Python 3.12).
  • Full CI test selection (tests/ minus the four ignored files, -m "not integration and not network and not docker"), run in parallel: 6383 passed, 122 skipped, 0 failed. The server-based test_maze_environment.py (needs gunicorn) and the legacy test_connect4_env.py are among the skipped tests.
  • bash .claude/hooks/lint.sh, ruff check, ruff format --check on the changed files, and python scripts/sync_env_docs.py --check pass. usort check flags tests/envs/test_grid_world.py, which it already flags on main (listed in AGENTS.md).

Claude Code Review

Alignment review by Opus 5.5, in the format of .claude/skills/alignment-review/SKILL.md.

Automated Checks

  • Lint: PASS, bash .claude/hooks/lint.sh
  • Debug code: CLEAN. The diff adds no print, breakpoint or TODO lines.

Open RFCs Context

No open RFC covers per-environment episode termination. RFC 014 and #1507 cover episode limits for direct MCP calls, which these four non-MCP environments don't have.

Tier 1: Fixes Required

None.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. The reset/step/state signatures are unchanged, rewards stay inside each environment, and reset() still starts a clean episode.

RFC Conflicts

None identified.

Summary

  • 0 mechanical issues to fix
  • 0 alignment points for human review
  • 0 RFC conflicts to discuss

Note

Medium Risk
Changes core step() semantics for four RL environments, which can affect training loops that step past done; behavior is intentional and aligned with other OpenEnv envs but is a correctness fix with broad test coverage.

Overview
Chess, Connect4, Grid World, and Maze now treat a finished episode as closed until reset(): each environment keeps a terminal _final_observation and step() returns a deep copy of it with done=True, the same reward/result, and no further moves or scoring (including duplicate goal/exit payouts and Connect4/Chess incorrectly continuing play).

Documentation in the env READMEs and docs/source/environments/ describes this post-terminal step() behavior.

New tests cover steps after win/loss/checkmate/stalemate, invalid Connect4 moves, and observation copying in maze.

Reviewed by Cursor Bugbot for commit 71b22a5. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a34a9c6. Configure here.

A custom FEN can start the game in checkmate or stalemate, so reset()
already returns done=True. Store that observation like a terminal step
does, so step() returns it again instead of scoring a move.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant