Skip to content

feat: add Scanner.WithScannerBufferSize for games over the token limit - #133

Merged
CorentinGS merged 3 commits into
mainfrom
fix/v2-scanner-buffer-size
Oct 25, 2026
Merged

CorentinGS merged 3 commits into
mainfrom
fix/v2-scanner-buffer-size

Conversation

@CorentinGS

Copy link
Copy Markdown
Owner

Problem

bufio.Scanner rejects any token larger than 64 KiB, and the scanner's split function returns one whole game per token. A game carrying large comments or variations therefore cannot be read at all:

HasNext: false
LastError: bufio.Scanner: token too long

This is exactly the case reported in #129: the longest possible chess game (17697 moves, 131 KB) fails with bufio.Scanner: token too long.

Fix

WithScannerBufferSize raises the scanner's maximum token size, mirroring WithPGNBufferSize on the v3 branch:

scanner := chess.NewScanner(f, chess.WithScannerBufferSize(16*1024*1024))

The default is unchanged (64 KiB), so this is opt-in and does not change behaviour for existing callers.

With the option, the longest possible chess game parses:

HasNext: true
ParseNext: moves=17697 outcome=1-0

Tests

TestScannerBufferSizeAllowsLargeGames (new): a game over the default limit is rejected by the default scanner and read successfully with WithScannerBufferSize.

go test ./... passes.

Note

This is the fix for the cause of the failure #129 reports. #129 makes the error visible (Scanner.LastError()); this makes the game readable. They are independent and can land in either order.

@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.12%. Comparing base (bca363e) to head (6b543b0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #133      +/-   ##
==========================================
- Coverage   81.14%   81.12%   -0.02%     
==========================================
  Files          28       28              
  Lines        4470     4477       +7     
==========================================
+ Hits         3627     3632       +5     
  Misses        716      716              
- Partials      127      129       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Corentin GS and others added 2 commits October 25, 2026 10:55
bufio.Scanner rejects a token larger than 64 KiB, so the scanner could not
read a game carrying large comments or variations — the longest possible
chess game (17697 moves) fails with "bufio.Scanner: token too long".
WithScannerBufferSize raises the limit for callers that need it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CorentinGS
CorentinGS force-pushed the fix/v2-scanner-buffer-size branch from facec9d to 045bccc Compare October 25, 2026 09:56
The option now clamps the size to bufio's default instead of the scanner
checking it later, and NewScanner always sets the buffer, so no new branch
is introduced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CorentinGS
CorentinGS merged commit c5e0038 into main Oct 25, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant