MismatchedTokenError: '#' inside a continuation part of an implicitly-concatenated f-string

[stevemadere]

Describe the bug

patchedast.get_patched_ast raises MismatchedTokenError (or, depending on the exact source shape, AttributeError/ValueError) when a literal # character appears inside the non-first part of an implicitly-concatenated f-string. Python's own tokenizer and ast.parse handle this fine — confirmed directly (see below) — so this looks like a bug in patchedast's own scanning, not in the AST it's patching.

To Reproduce

from rope.refactor import patchedast

text = '''a = 1
E(
    f"{a} one "
    f"#52 two")
'''
patchedast.get_patched_ast(text, True)

Traceback:

Traceback (most recent call last):
  File ".../rope/refactor/patchedast.py", line 837, in consume
    new_offset = self.source.index(token, self.offset)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: substring not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File ".../rope/refactor/patchedast.py", line 843, in consume
    raise MismatchedTokenError(
rope.refactor.patchedast.MismatchedTokenError: Token <)> at (4, 15) cannot be matched

Isolating the trigger

Varied one thing at a time against a real codebase that writes long, wrapped diagnostic messages as implicitly-concatenated f-strings (often citing issue numbers, hence the frequent #):

shape	result
# in a continuation part of an f-string concatenation	BREAKS
same, # removed	ok
same, plain strings instead of f-strings (no f prefix)	ok
# in the first part only	ok
single f-string containing # (no concatenation)	ok
assignment target rather than a call argument	BREAKS — so it isn't specific to call syntax

All of these parse fine under plain ast.parse — only patchedast fails.

What I think is happening

Tokenizing the repro directly (Python 3.12, PEP 701-style f-string tokens) shows the # is correctly tokenized as literal FSTRING_MIDDLE content, never as a comment:

FSTRING_START  'f"'   (2, 4)-(2, 6)
OP             '{'    ...
NAME           'a'    ...
OP             '}'    ...
FSTRING_MIDDLE ' one ' ...
FSTRING_END    '"'    ...
NL             '\n'   ...
FSTRING_START  'f"'   (3, 4)-(3, 6)
FSTRING_MIDDLE '#52 two' (3, 6)-(3, 13)
FSTRING_END    '"'    ...
OP             ')'    ...

So Python's real tokenizer has no trouble with this. patchedast's own scanning appears to consume the first f-string literal from the AST correctly, then resume scanning the remaining source as raw text for the rest of the expression — and in that fallback, an unescaped # opens what looks like a real comment, swallowing the rest of the line (including the closing paren consume() is looking for). That would explain both the MismatchedTokenError here and the ValueError: substring not found it's chained from.

Environment

• rope version: 1.14.0
• Python version: 3.12.13
• OS: macOS (Darwin)

Related, but distinct, existing issues (for context — none of these appear to be the same trigger):

• patched_ast will set region[0] to None when unmatched parenthesis inside string #499 — a stray unmatched ) written inside a single f-string's own literal text sets region[0] to None. Different symptom, different trigger (no concatenation involved).
• Rope exception when f-string has a # before a variable when using extract method #807 — f"#{h}" breaks extract's own similarfinder pattern-matching (a re.sub + ast.parse of an extracted pattern string, not patchedast on the real source). Different code path entirely.
• SyntaxError when parsing f-string with Python 3.12 new syntax #815 — nested quotes in a single f-string break occurrences.py's own regex-based textual search. Also unrelated to patchedast.

Happy to try to narrow this further or test a patch if that's useful — the four-line repro above should make it easy to reproduce directly.