Skip to content

Fix patchedast region for implicitly-concatenated f-strings (fixes #879) - #880

Open
stevemadere wants to merge 1 commit into
python-rope:masterfrom
stevemadere:fix-879-fstring-implicit-concatenation-region
Open

stevemadere wants to merge 1 commit into
python-rope:masterfrom
stevemadere:fix-879-fstring-implicit-concatenation-region

Conversation

@stevemadere

Copy link
Copy Markdown

Description

Fixes an undocumented string parsing crash-inducing bug in _PatchingASTWalker._JoinedStr()

_JoinedStr builds its children as [start_quote, *FormattedValues, end_quote] and lets _handle consume each by forward search. The end-quote child therefore matches the first closing quote after the last interpolation -- which, in an implicit concatenation, closes only the part that held that interpolation, not the whole concatenation.

The node's region was then understated and, worse, the scanner was left parked inside source text that had not been recognised as a string literal. A # in a trailing part -- f"{a} one " f"#52 two" -- then reads to _good_token as the start of a real comment, so the following ) is skipped as commented-out and consume raises MismatchedTokenError (issue #879).

The truncation itself predates the #: a hash-free concatenation such as f"{a} one " f"52 two" also produced a short region, it just failed silently because nothing downstream needed to match a token past the truncation point.

consume_string already measures the full concatenation before _JoinedStr rewinds the scanner, so close the gap from its end offset rather than re-scanning. Plain (non-f) strings were never affected because _Str consumes via that same pattern.

Fixes #879

Checklist (delete if not relevant):

  • I have added tests that prove my fix is effective or that my feature works
  • I have updated CHANGELOG.md
  • The documentation already implied it worked this way by neglecting to mention the crash. ;-)

`_JoinedStr` builds its children as [start_quote, *FormattedValues,
end_quote] and lets `_handle` consume each by forward search. The
end-quote child therefore matches the first closing quote after the last
interpolation -- which, in an implicit concatenation, closes only the
part that held that interpolation, not the whole concatenation.

The node's region was then understated and, worse, the scanner was left
parked inside source text that had not been recognised as a string
literal. A `#` in a trailing part -- `f"{a} one " f"python-rope#52 two"` -- then
reads to `_good_token` as the start of a real comment, so the following
`)` is skipped as commented-out and `consume` raises
MismatchedTokenError (issue python-rope#879).

The truncation itself predates the `#`: a hash-free concatenation such
as `f"{a} one " f"52 two"` also produced a short region, it just failed
silently because nothing downstream needed to match a token past the
truncation point.

`consume_string` already measures the full concatenation before
`_JoinedStr` rewinds the scanner, so close the gap from its end offset
rather than re-scanning. Plain (non-f) strings were never affected
because `_Str` consumes via that same pattern.

Fixes python-rope#879

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MismatchedTokenError: '#' inside a continuation part of an implicitly-concatenated f-string

1 participant