Conversation
The statement splitter treats a newline after ';' as the start of a new
statement, but a trailing run with nothing after it became a
whitespace-only statement that the final guard dropped. So
parse('SELECT 1;\n') lost the newline while parse('SELECT 1;\t') did
not, breaking the parse/str round-trip.
Hold the previous statement back by one step and re-attach a final
whitespace-only run to it. Whitespace-only input still yields no
statements, and statement counts are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sqlparse.parse()drops a trailing whitespace run that follows the final;when that run contains a newline, breaking the parse → str round-trip:Cause
In
StatementSplitter.process, a;setsconsume_ws = True. A following newline is deliberately treated as not whitespace (so the next statement starts at the newline), so it triggers a split and begins a fresh statement. When nothing but whitespace follows to end-of-input, that fresh statement is whitespace-only, and the final guarddiscards it — taking the newline with it. A trailing tab or spaces stays in
EOS_TTYPE, so it never splits and is preserved; only newlines are lost, which is why the two cases disagree.Fix
Hold the previous statement back by one step, and at end-of-input re-attach a final whitespace-only run to it instead of dropping it. Whitespace-only input (
parse(" ")) still yields no statements, and statement counts for every existing case are unchanged.Added
test_split_trailing_whitespace_preserved(round-trip for trailing spaces/tabs/newlines) andtest_split_whitespace_only; the newline cases fail onmaster. Full suite passes (506).