Refactor SQLovely internals

.github/workflows/ci.yml

@@ -19,9 +19,7 @@ jobs:
           cache: npm
       - name: Install dependencies
         run: npm ci
-      - name: Type check
-        run: npm run check
-      - name: Test
-        run: npm test
+      - name: Validate
+        run: npm run validate
       - name: Package VSIX
-        run: npm run package:vsix
+        run: node ./scripts/packageVsix.js

.prettierignore

@@ -0,0 +1,7 @@
+node_modules
+out
+dist
+coverage
+*.vsix
+*.zip
+syntaxes/*.json

.prettierrc

@@ -0,0 +1,5 @@
+{
+  "singleQuote": true,
+  "trailingComma": "all",
+  "printWidth": 100
+}

CHANGELOG.md

@@ -2,6 +2,17 @@
 
 ## Unreleased
 
+- Streamlined CI validation to run the combined validation workflow once before packaging the VSIX.
+- Added explicit Prettier formatting, ESLint linting and TypeScript typecheck scripts to the project validation workflow.
+- Isolated loose legacy metadata-header aliases, candidate parsing and migration logic into focused internal modules.
+- Extracted formatter indentation handling into a dedicated indentation engine while preserving existing formatter output.
+- Documented the internal formatter pipeline design, pass order and rules for adding future formatter passes.
+- Organized formatter regression tests into topic-based suites under `test/formatter/` while keeping the existing formatter test command as the entry point.
+- Split metadata-header parsing, rendering, history synchronization and placement helpers into dedicated internal modules without changing generated metadata behavior.
+- Debounced SQL diagnostics on document changes and reused formatter safety limits to skip expensive metadata diagnostics for very large SQL documents.
+- Centralized formatter options, indentation settings, cancellation checks and safety decisions in a shared formatting context.
+- Grouped formatter passes into structural and cleanup phases to clarify rule ownership without changing formatter behavior.
+- Refactored Watcom formatter expansion passes behind an explicit internal pipeline while preserving existing formatter output.
 - Fixed Watcom DDL/list parenthesis cleanup so temporary-table closing parentheses align with the declaration and safe trailing DDL commas before `)` are removed.
 - Fixed Watcom statement continuation cleanup for multiline `UPDATE ... SET` assignments, compact `SELECT` comma spacing and arithmetic operator spacing outside strings/comments.
 - Fixed split Watcom `ORDER BY IF ... ENDIF` expression continuations so following sort keys keep their comma separator.
@@ -50,7 +61,6 @@
 - SQLovely formatting can now apply enabled SQLovely Extras such as metadata-header insertion or updates.
 - Updated documentation and example workspace settings for the new formatting/extras behavior.
 
-
 ## 0.1.4
 
 - Renamed optional file-updating features to SQLovely Extras.

README.md

@@ -69,7 +69,6 @@ SQLovely formats conservatively. It keeps SQL structure intact and focuses on re
 
 Apart from normalizing compact Watcom `IF ... THEN ... END IF` control-flow statements, preserving and normalizing expression-style `IF ... THEN ... ELSE ... ENDIF` constructs, keeping `UNION ALL` on its own line, splitting Watcom parenthesized argument/parameter lists across indented lines, placing top-level Watcom query clauses on stable physical lines with continuation indentation, cleaning multiline `UPDATE ... SET` assignments plus compact comma/operator spacing and DDL/list closing parentheses, recovering separators after split `ORDER BY IF ... ENDIF` expression continuations, indenting cursor `FOR ... DO` loops, formatting Watcom `CASE` expressions and aligning Watcom exception handlers, splitting/counting stacked Watcom block endings before indentation, and treating `ELSEIF` as a branch keyword instead of a nested block opener, the formatter does not perform schema-aware rewrites or migrate SQL between dialects.
 
-
 ### Formatter safety guards
 
 SQLovely includes safety guards for very large generated SQL files. When a document exceeds the configured safety limits, expensive Watcom rewrite passes such as query-clause splitting, parenthesis splitting, cursor-loop splitting, CASE formatting and compact IF expansion are skipped. Lightweight cleanup such as keyword casing, whitespace cleanup and indentation still runs.

docs/DEVELOPMENT.md

@@ -4,6 +4,7 @@ This document contains the development workflow for SQLovely.
 
 For user-facing extension usage, see `README.md`.
 For release packaging, see `PACKAGING.md`.
+For formatter architecture details, see `docs/FORMATTER_PIPELINE.md`.
 
 ## Setup
 
@@ -21,12 +22,26 @@ npm ci
 
 ## Validate
 
-Run TypeScript checks:
+Run the full static validation workflow:
 
 ```bash
 npm run check
 ```
 
+This runs the TypeScript typecheck, ESLint and the Prettier formatting check. The checks can also be run individually:
+
+```bash
+npm run typecheck
+npm run lint
+npm run format:check
+```
+
+Format source files with Prettier:
+
+```bash
+npm run format
+```
+
 Run the test suite:
 
 ```bash
@@ -57,6 +72,7 @@ Useful smoke tests:
 - verify split `ORDER BY IF ... ENDIF` expression continuations keep the comma before the next sort key
 - run **SQLovely: Insert or Update Metadata Header**
 - verify diagnostics and quick fixes
+- verify diagnostics update after a short edit debounce and stay responsive on large SQL files
 - verify metadata headers in a script with multiple procedures, functions or triggers
 - verify loose legacy metadata headers are normalized only when they contain a recognizable version field
 - verify long metadata descriptions are wrapped without removing manual line breaks
@@ -103,6 +119,10 @@ For MSSQL-oriented smoke tests:
 
 It intentionally does not apply SQLovely Extras. Keep this behavior conservative because the command can touch many files at once. Directory formatting forwards VS Code cancellation requests into the formatter and uses the same safety guards as normal document formatting.
 
+## Diagnostics performance behavior
+
+SQLovely diagnostics run immediately when SQL documents are opened and when relevant configuration changes. Text-change diagnostics are debounced so large SQL files are not reparsed on every keystroke. Missing-metadata diagnostics reuse the formatter safety limits and are skipped for documents that exceed the complex-document thresholds; max-line-length diagnostics remain available because they are lightweight and line-oriented.
+
 ## Metadata-header regression focus
 
 When changing metadata-header behavior, add or update regression tests for:
@@ -115,6 +135,35 @@ When changing metadata-header behavior, add or update regression tests for:
 - date normalization, including two-digit legacy years
 - multiline description wrapping and manual line-break preservation
 
+## Metadata-header internals
+
+Metadata-header behavior is split into small internal modules under `src/extras/metadata/`:
+
+- `metadataHeaderParser.ts` locates current, legacy and loose legacy headers and parses fields/history entries.
+- `metadataHeaderRenderer.ts` renders the current compact `-- METADATA` header layout.
+- `metadataHistory.ts` synchronizes version and history entries.
+- `metadataHeaderPlacement.ts` decides where headers are inserted relative to the detected SQL object.
+- `metadataText.ts` owns shared text-range and line-boundary helpers.
+- `legacyMetadataAliases.ts` owns loose legacy label aliases and field classifiers.
+- `legacyMetadataParser.ts` finds loose legacy header candidates around SQL objects and normalizes comment text.
+- `legacyMetadataMigration.ts` maps loose legacy fields and history entries to the modern metadata model.
+
+Keep `src/extras/metadataHeader.ts` as the public orchestration entry point. New metadata behavior should usually live in the focused helper module that owns that concern, with regression coverage in the metadata test suites.
+
+## Formatter test organization
+
+Formatter regression coverage is grouped by topic under `test/formatter/`. Keep `test/runFormatterTests.js` as the stable entry point for `npm run test:formatter`, and add new formatter regressions to the closest topic suite instead of expanding the runner directly. Use `test/formatter/helpers.js` for shared formatter imports, dialects, default options and fixture loading.
+
+## Formatter pipeline internals
+
+Watcom structural rewrites are coordinated through `src/formatter/formattingPipeline.ts`. Shared formatter inputs such as the active dialect, resolved options, indentation string, cancellation checks and safety decision live in `src/formatter/formattingContext.ts`. Indentation is isolated in `src/formatter/indentation/indentationEngine.ts`, which applies keyword casing, block depth, query continuation depth, parenthesis continuation depth and CASE/exception branch depth after structural expansion. Keep the pipeline order explicit and behavior-preserving: compact/control-flow expansion, query/cursor/exception/expression normalization, block-ending normalization and parenthesis splitting should run before the indentation engine and final cleanup passes. When adding a formatter rule, prefer a small stateful pipeline pass that consumes the shared formatting context instead of expanding formatter orchestration logic in `formatSql.ts`.
+
+Formatter pass files are grouped by phase:
+
+- `src/formatter/passes/structural/` contains rewrite passes that can split or reshape SQL lines before indentation.
+- `src/formatter/passes/cleanup/` contains narrow line-level cleanup passes that run after indentation and should not introduce new structural SQL splits.
+
+See `docs/FORMATTER_PIPELINE.md` for the full pass order, invariants and checklist for adding formatter rules.
 
 ## Formatter performance regression focus
 

docs/FORMATTER_PIPELINE.md

@@ -0,0 +1,82 @@
+# Formatter Pipeline Design
+
+This document describes the internal SQLovely formatter pipeline. It is intended for maintainers who add, move or debug formatter rules.
+
+The formatter is deliberately organized as a behavior-preserving pipeline rather than one large pass. Watcom SQL has several compact constructs that can be written on one physical line, and later indentation logic is much simpler when those constructs are normalized first.
+
+## High-level flow
+
+`formatSql` coordinates formatting in this order:
+
+1. Split the source text into physical lines while preserving the original end-of-line style.
+2. Create a shared `FormattingContext` with the active dialect, resolved options, safety decision, indentation string and cancellation helpers.
+3. Run structural expansion through `createFormattingPipeline`.
+4. Apply keyword casing and indentation to the expanded lines through the dedicated indentation engine.
+5. Run narrow cleanup passes for line-level fixes that depend on final indentation.
+6. Rejoin the lines and restore the final newline policy.
+
+The pipeline should keep this separation intact: structural passes may split or reshape SQL before indentation; cleanup passes should only repair narrow final-line shapes and should not introduce broad SQL structure changes.
+
+## Structural pass order
+
+Structural passes live under `src/formatter/passes/structural/` and run before indentation. The current Watcom pipeline order is:
+
+1. `inlineIfFormatting` expands compact procedural `IF ... THEN ... END IF` blocks.
+2. `unionAllFormatting` keeps `UNION ALL` on a stable physical line.
+3. `cursorForFormatting` normalizes `FOR ... CURSOR FOR ... DO ... END FOR` loops.
+4. `queryClauseFormatting` splits top-level query clauses and predicate continuations.
+5. `exceptionFormatting` normalizes `EXCEPTION` / `WHEN ... THEN` handlers.
+6. `ifExpressionFormatting` preserves scalar Watcom `IF ... THEN ... ELSE ... ENDIF` expressions.
+7. `caseExpressionFormatting` normalizes `CASE ... WHEN ... THEN ... ELSE ... END` expressions.
+8. `blockEndFormatting` splits stacked block endings such as `END IF END IF;`.
+9. `parenthesisFormatting` splits multiline parenthesized argument, parameter and list shapes.
+
+Pass order matters. For example, procedural IF expansion must happen before IF-expression preservation, query clause splitting must happen before indentation, and stacked block endings must be split before block-depth accounting.
+
+## Indentation stage
+
+`src/formatter/indentation/indentationEngine.ts` owns indentation because it has to combine several independent depth sources:
+
+- procedural block depth (`BEGIN`, `IF`, `WHILE`, `FOR`, `LOOP`, `ELSE`, `END`, etc.)
+- query continuation depth (`SELECT`, `INTO`, `ORDER BY`, `GROUP BY`, logical predicates)
+- parenthesis continuation depth
+- scalar `CASE` expression depth
+- exception-handler branch depth
+
+Do not add broad structural rewrites directly to the indentation engine. Add them as a structural pass first, then adjust indentation only when the new structure needs a new depth rule. Keep `formatSql.ts` focused on orchestration, cleanup and final text assembly.
+
+## Cleanup pass order
+
+Cleanup passes live under `src/formatter/passes/cleanup/` and run after indentation. They intentionally handle narrow final-shape issues, such as:
+
+- multiline `UPDATE ... SET` assignment continuations
+- safe comma and arithmetic spacing outside strings/comments
+- DDL/list closing-parenthesis alignment
+- safe removal of trailing commas immediately before DDL/list closing parentheses
+
+Cleanup passes should not perform large rewrites. If a rule needs to split SQL statements, track strings/comments across lines, or change block structure, it belongs in the structural pipeline instead.
+
+## Safety and cancellation
+
+Every formatter entry point creates a `FormattingContext`. Structural passes should use the context instead of creating independent safety decisions. The pipeline already applies `shouldRunExpensiveLineFormatting` so expensive passes are skipped for very large documents or very long physical lines.
+
+Formatter code should preserve these invariants:
+
+- cancellation returns the original text unchanged
+- large-document guards skip expensive rewrites but keep lightweight cleanup active
+- long-line guards avoid complex line-level transformations
+- scanner state remains valid even when a pass is skipped
+
+## Adding a formatter rule
+
+Use this checklist for new rules:
+
+1. Decide whether the rule is structural or cleanup-only.
+2. Add the rule to the appropriate `passes/structural` or `passes/cleanup` folder.
+3. Keep scanning lexical and linear; do not use broad regular expressions across whole documents when a line scanner is enough.
+4. Preserve strings, quoted identifiers and comments.
+5. Thread any cross-line state through a small explicit state object.
+6. Add focused regression tests under `test/formatter/`.
+7. Update this document if the pass order or formatter invariants change.
+
+When in doubt, prefer a small pass with one responsibility over adding more conditional logic to `formatSql.ts`.