Skip to content

Preserve trailing whitespace after the final semicolon - #908

Open
itzzdev09 wants to merge 1 commit into
andialbrecht:masterfrom
itzzdev09:fix-trailing-whitespace-dropped
Open

itzzdev09 wants to merge 1 commit into
andialbrecht:masterfrom
itzzdev09:fix-trailing-whitespace-dropped

Conversation

@itzzdev09

Copy link
Copy Markdown

sqlparse.parse() drops a trailing whitespace run that follows the final ; when that run contains a newline, breaking the parse → str round-trip:

>>> import sqlparse
>>> "".join(str(s) for s in sqlparse.parse("SELECT 1;\n"))
'SELECT 1;'          # the "\n" is gone
>>> "".join(str(s) for s in sqlparse.parse("SELECT 1;\t"))
'SELECT 1;\t'        # a tab is kept, so the two disagree

Cause

In StatementSplitter.process, a ; sets consume_ws = True. A following newline is deliberately treated as not whitespace (so the next statement starts at the newline), so it triggers a split and begins a fresh statement. When nothing but whitespace follows to end-of-input, that fresh statement is whitespace-only, and the final guard

if self.tokens and not all(t.is_whitespace for t in self.tokens):

discards it — taking the newline with it. A trailing tab or spaces stays in EOS_TTYPE, so it never splits and is preserved; only newlines are lost, which is why the two cases disagree.

Fix

Hold the previous statement back by one step, and at end-of-input re-attach a final whitespace-only run to it instead of dropping it. Whitespace-only input (parse(" ")) still yields no statements, and statement counts for every existing case are unchanged.

Added test_split_trailing_whitespace_preserved (round-trip for trailing spaces/tabs/newlines) and test_split_whitespace_only; the newline cases fail on master. Full suite passes (506).

The statement splitter treats a newline after ';' as the start of a new
statement, but a trailing run with nothing after it became a
whitespace-only statement that the final guard dropped. So
parse('SELECT 1;\n') lost the newline while parse('SELECT 1;\t') did
not, breaking the parse/str round-trip.

Hold the previous statement back by one step and re-attach a final
whitespace-only run to it. Whitespace-only input still yields no
statements, and statement counts are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant