Repository navigation
perf(python): reduce temporal conversion column allocations - #169
Merged
Chenglong Wang (Chenglong-MS) merged 4 commits intoNov 15, 2026
Merged
Chenglong Wang (Chenglong-MS) merged 4 commits into
Chenglong Wang (Chenglong-MS) merged 4 commits into
Conversation
Member
Author
|
Chenglong Wang (@Chenglong-MS), could you please review this PR? Thanks! |
Contributor
|
Looking good, I'll merge them to dev and merge all to main as the 1.0.2 release. I've retargeted this PR to |
This was referenced Nov 15, 2026
Merged
Chenglong Wang (Chenglong-MS)
added a commit
that referenced
this pull request
Nov 16, 2026
* Title wrap: fullest lines with no widow, plus a comparison page
A title or deck that ends in a widow is broken again over the same number
of lines, as full as they can be: a DP charges every line but the last its
squared room left on the block, and the last must reach a third of it.
Replaces evening the last two lines, which left one full line over two
half lines. Playground page #/playground/title-wrap compares the rules.
* feat(mcp): validate and show report only what to act on
The report no longer repeats the spec (chart type, theme, encodings,
interactions, "User interactions: none."). Each chart lists its errors,
warnings and update ops that will not apply; a finding several charts
share is listed once; the layout takes one line. Link notes on linked
charts sit apart and do not count as something to fix; the shared colour
order note is gone. A clean spec reads "Nothing to fix: call show with
this spec." validate_chart returns the text report alone for vegalite,
so each warning appears once. The skills describe the new report.
* fix(table): only the rows scroll, the table draws its theme's furniture, and a source says its lead-in once
FlintTable keeps the title, subtitle and notes in place and scrolls the
rows under sticky column headings. A chart's table carries the plan's
header pieces, closing rules and source line, and FlintTable draws them,
so a table under the Economist or BBC theme gets its tab, rules, source
and house credit. A source that already starts with the theme's lead-in
("Source: Gapminder") prints it once. The card's table PNG draws the
rules, the tab and the source line.
* fix some layout issue
* add navigator and fix talk slides
* updates
* layout algorithm update
* updates
* fixes
* fix(site): keep the page scrolling under embedded MCP cards
ChartView locks overflow on html and body for its own iframe; the site embeds
it in the page itself, so the playground gallery could not scroll. Call
useHostPage above the card in the three embeds that lacked it.
* bug fixes
* fixes
* fix(layout): hold a Vega-Lite drawing within its canvasSize
The canvas-fit pass judged a chart by how far the host would scale it, and left
one that read well drawn up to ~30% larger than its box: keys beside a plot that
filled the width, a title over one that filled the height. Hosts that draw the
SVG at its own size (Data Formulator) cut them off.
After the fit, planLayout now runs containInCanvas: where an explicit
canvasSize is given and the measured drawing runs past it, the chart is laid
out again for a smaller room (CanvasTreatment.contain) - the fit's edits kept,
keys held at their edge - found by a few rounds of bisection per side. A round
is kept only if the plot keeps >= 50% of each side, no more values are cut by
overflow truncation and no label is turned that stood flat; a result that is
nearer but not within must gain at least min(3 points, half the gap) of scale.
Reported as 'canvas-contain'; skipped for canvasFit 'off', composites, pinned
sessions replays (replayed via the treatment), and unmeasured furniture.
The drawing estimate used by this step also counts a deck printed without a
headline (furniture-reserve deckAlone), a row of keys from its anchor along the
plot, the half of a top y label that stands above the plot, and leaves out
series-end names that are drawn inside the plot. These are scoped to the new
step: charts without canvasSize lay out byte-identically (5441 corpus cases).
Tests: canvas-contain.test.ts renders the issue's charts (stacked, line, donut,
bar) under none / keys-on-top / nyt / datawrapper at 600x300, 290x300, 290x240
and asserts the SVG stays within the box. row-names-above: the note now names
the room laid out for. live-values: the pushed-vs-fresh comparison tolerates
<=1px at the root frame only - a pre-existing Vega incremental-layout quirk
(8/76 transitions at baseline) that the new sizes happen to hit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
* test(vegalite): compile value-sorted bars at every size under every house
Data Formulator hit "Cannot destructure property 'aggregate' of
'fieldDefToSortBy'" in flint-chart 1.0.1: datawrapper sets row names above the
bars only in narrow, tall boxes (240x420, 320x420 for 14 long names), and the
channel sort "-x" on that layer, which has no x, threw inside Vega-Lite. The
fix is already on dev (bandOrder / alignBandOrders, ff18201 + aad6b2d); the
existing test covered one size.
The new grid compiles 5 sorted-bar variants (sortBy x desc/asc, columns sorted
by y, field sort, stacked) at 6 widths x 4 heights under no theme and every
THEME_PRESETS house with vega-lite compile, asserting no throw and no dropped
sort ("conflicting sort properties"), and checks the grid reaches the
row-names-above layer at 240x420 and 320x420. It fails under 16 houses with
alignBandOrders disabled.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
* fix(interactive): shape filtered rows as the chart's data was shaped
filterControls swaps the rows a filter leaves into the compiled inline
source through prepareDataRows, which only converted temporal values. The
compiled data of a chart with an aggregate encoding (or several y fields)
holds rows Flint folded and aggregated at build time, so the raw rows had
no price_average-style column and the chart emptied. prepareDataRows now
uses the session's shapeRows, the same analysis step that built the data
and that replan already uses for filtered rows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
* layout fixes, binning color etc
* 1.0.2
* annotation, layout, maps
* ok
* fixes and talk
* fix interaction, annotation, box overflow, and tooltip issues
* Roll compact numbers over to the next unit when rounding reaches 1000 (#172)
999,600 printed as 1000k (or 1,000k under one unit per set) because the
unit was chosen from the unrounded magnitude. The unit now moves up where
the value rounds to 1000 at the precision it prints with, in the JS
formatter, the Vega expression, the shared-unit path, the KPI tile and the
label-width estimate.
Fixes #172
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updates dev and fixes
* perf(js): lazily discover fields for encoding sort repair (#168)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* perf(js): lazily discover fields for encoding sort repair
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf(python): reduce temporal conversion column allocations (#169)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* Reduce temporal conversion column allocations
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf(python): avoid preliminary full semantics pass (#170)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* Avoid preliminary full Python semantics pass
* Preserve Python type pass benchmark evidence
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf: add reproducible compiler and MCP render baselines (#171)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* Add reproducible compiler and render baselines
* Bound render benchmark workers and guarantee cleanup
* Record bounded local render baseline
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf: index explicit overflow order membership (#167)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* perf: index explicit overflow order membership
Replace per-entry linear membership scans in the default overflow
explicit-value order (JS defaultOverflowOrder, Python
_default_overflow_strategy) with an invocation-local Set.
JS Set.has uses the same SameValueZero equality as Array.includes.
Python set membership matches list membership for the hashable
unique values (incl. True == 1); unhashable JSON entries fall back
to list membership so they are still skipped rather than raising.
Adds scripts/bench-overflow-order (harness + recorded 3-round results).
Refs #161
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* slides update
* prep release
* updates
* fix
---------
Co-authored-by: xavier-shaw <xshaw2002@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Shayne Boyer <7681382+spboyer@users.noreply.github.com>
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
Chenglong Wang (Chenglong-MS)
added a commit
that referenced
this pull request
Nov 16, 2026
* Title wrap: fullest lines with no widow, plus a comparison page
A title or deck that ends in a widow is broken again over the same number
of lines, as full as they can be: a DP charges every line but the last its
squared room left on the block, and the last must reach a third of it.
Replaces evening the last two lines, which left one full line over two
half lines. Playground page #/playground/title-wrap compares the rules.
* feat(mcp): validate and show report only what to act on
The report no longer repeats the spec (chart type, theme, encodings,
interactions, "User interactions: none."). Each chart lists its errors,
warnings and update ops that will not apply; a finding several charts
share is listed once; the layout takes one line. Link notes on linked
charts sit apart and do not count as something to fix; the shared colour
order note is gone. A clean spec reads "Nothing to fix: call show with
this spec." validate_chart returns the text report alone for vegalite,
so each warning appears once. The skills describe the new report.
* fix(table): only the rows scroll, the table draws its theme's furniture, and a source says its lead-in once
FlintTable keeps the title, subtitle and notes in place and scrolls the
rows under sticky column headings. A chart's table carries the plan's
header pieces, closing rules and source line, and FlintTable draws them,
so a table under the Economist or BBC theme gets its tab, rules, source
and house credit. A source that already starts with the theme's lead-in
("Source: Gapminder") prints it once. The card's table PNG draws the
rules, the tab and the source line.
* fix some layout issue
* add navigator and fix talk slides
* updates
* layout algorithm update
* updates
* fixes
* fix(site): keep the page scrolling under embedded MCP cards
ChartView locks overflow on html and body for its own iframe; the site embeds
it in the page itself, so the playground gallery could not scroll. Call
useHostPage above the card in the three embeds that lacked it.
* bug fixes
* fixes
* fix(layout): hold a Vega-Lite drawing within its canvasSize
The canvas-fit pass judged a chart by how far the host would scale it, and left
one that read well drawn up to ~30% larger than its box: keys beside a plot that
filled the width, a title over one that filled the height. Hosts that draw the
SVG at its own size (Data Formulator) cut them off.
After the fit, planLayout now runs containInCanvas: where an explicit
canvasSize is given and the measured drawing runs past it, the chart is laid
out again for a smaller room (CanvasTreatment.contain) - the fit's edits kept,
keys held at their edge - found by a few rounds of bisection per side. A round
is kept only if the plot keeps >= 50% of each side, no more values are cut by
overflow truncation and no label is turned that stood flat; a result that is
nearer but not within must gain at least min(3 points, half the gap) of scale.
Reported as 'canvas-contain'; skipped for canvasFit 'off', composites, pinned
sessions replays (replayed via the treatment), and unmeasured furniture.
The drawing estimate used by this step also counts a deck printed without a
headline (furniture-reserve deckAlone), a row of keys from its anchor along the
plot, the half of a top y label that stands above the plot, and leaves out
series-end names that are drawn inside the plot. These are scoped to the new
step: charts without canvasSize lay out byte-identically (5441 corpus cases).
Tests: canvas-contain.test.ts renders the issue's charts (stacked, line, donut,
bar) under none / keys-on-top / nyt / datawrapper at 600x300, 290x300, 290x240
and asserts the SVG stays within the box. row-names-above: the note now names
the room laid out for. live-values: the pushed-vs-fresh comparison tolerates
<=1px at the root frame only - a pre-existing Vega incremental-layout quirk
(8/76 transitions at baseline) that the new sizes happen to hit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
* test(vegalite): compile value-sorted bars at every size under every house
Data Formulator hit "Cannot destructure property 'aggregate' of
'fieldDefToSortBy'" in flint-chart 1.0.1: datawrapper sets row names above the
bars only in narrow, tall boxes (240x420, 320x420 for 14 long names), and the
channel sort "-x" on that layer, which has no x, threw inside Vega-Lite. The
fix is already on dev (bandOrder / alignBandOrders, ff18201 + aad6b2d); the
existing test covered one size.
The new grid compiles 5 sorted-bar variants (sortBy x desc/asc, columns sorted
by y, field sort, stacked) at 6 widths x 4 heights under no theme and every
THEME_PRESETS house with vega-lite compile, asserting no throw and no dropped
sort ("conflicting sort properties"), and checks the grid reaches the
row-names-above layer at 240x420 and 320x420. It fails under 16 houses with
alignBandOrders disabled.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
* fix(interactive): shape filtered rows as the chart's data was shaped
filterControls swaps the rows a filter leaves into the compiled inline
source through prepareDataRows, which only converted temporal values. The
compiled data of a chart with an aggregate encoding (or several y fields)
holds rows Flint folded and aggregated at build time, so the raw rows had
no price_average-style column and the chart emptied. prepareDataRows now
uses the session's shapeRows, the same analysis step that built the data
and that replan already uses for filtered rows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
* layout fixes, binning color etc
* 1.0.2
* annotation, layout, maps
* ok
* fixes and talk
* fix interaction, annotation, box overflow, and tooltip issues
* Roll compact numbers over to the next unit when rounding reaches 1000 (#172)
999,600 printed as 1000k (or 1,000k under one unit per set) because the
unit was chosen from the unrounded magnitude. The unit now moves up where
the value rounds to 1000 at the precision it prints with, in the JS
formatter, the Vega expression, the shared-unit path, the KPI tile and the
label-width estimate.
Fixes #172
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updates dev and fixes
* perf(js): lazily discover fields for encoding sort repair (#168)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* perf(js): lazily discover fields for encoding sort repair
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf(python): reduce temporal conversion column allocations (#169)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* Reduce temporal conversion column allocations
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf(python): avoid preliminary full semantics pass (#170)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* Avoid preliminary full Python semantics pass
* Preserve Python type pass benchmark evidence
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf: add reproducible compiler and MCP render baselines (#171)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* Add reproducible compiler and render baselines
* Bound render benchmark workers and guarantee cleanup
* Record bounded local render baseline
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* perf: index explicit overflow order membership (#167)
* [deploy] fix plotly issue and mcp (#152)
* local
* updates
* fixes
* perf: index explicit overflow order membership
Replace per-entry linear membership scans in the default overflow
explicit-value order (JS defaultOverflowOrder, Python
_default_overflow_strategy) with an invocation-local Set.
JS Set.has uses the same SameValueZero equality as Array.includes.
Python set membership matches list membership for the hashable
unique values (incl. True == 1); unhashable JSON entries fall back
to list membership so they are still skipped rather than raising.
Adds scripts/bench-overflow-order (harness + recorded 3-round results).
Refs #161
---------
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: Chenglong Wang <chenglong.wang@microsoft.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* slides update
* prep release
* updates
* fix
* Fix out-of-memory flint-js build and flaky hull timing test (#175)
* build(flint-js): build declarations in their own pass
tsup ran the ESM, CJS and declaration builds at once; together they need
more than 8 GB and the ADO Flint MCP build agent (8 GB, no swap) ran out of
memory. `npm run build` now runs the bundles, then `tsup --dts-only`.
Output is unchanged; each pass peaks near 5.7 GB under a 6 GB heap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test: give the million-point hull timing a CI-safe budget
The single timed run took 1066 ms on a loaded runner and failed the 1.1
release CI and npm publish. 3 s still fails a quadratic hull by minutes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test: time the enclosure cost suite by median, with CI-sized budgets
The suite averaged every run (and timed the million-point hull once), so
one stall on a shared runner failed it: main's 1.1 CI read 53 ms against
40, 66 against 60 and 1251 against 1000, while dev's run of the same code
passed. Take the median of warmed runs, as the placement cost test does,
with bounds about ten times a laptop's median.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* update tests to avoid CI failure
* test: steady the placement, hull and tween tests under load
Three full suites at once (heavier than a CI runner) still failed the
placement median at 46-62 ms against 40, timed out the million-point hull
at vitest's 5 s, and drew a 60 ms viewport tween in one late frame.
Placement's bound is now about ten times a laptop's median like the
enclosure suite; the hull takes the median of five runs on 200,000 points;
the tween runs on vitest's fake clock, so its frames do not depend on load.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* site: add Flint 1.1 to the landing updates
Links the 1.1 release tag, and points the 0.5.0 entry at its real tag (0.5).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: xavier-shaw <xshaw2002@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Shayne Boyer <7681382+spboyer@users.noreply.github.com>
Co-authored-by: xavier <75410443+xavier-shaw@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Copilot-Session: dcd3da51-67ac-4e66-b402-bfab78750c39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to #163; part of #160.
Infer temporal columns individually instead of retaining every first-row column vector. Preserve complete deepcopy when conversion is required and original-input identity when it is not, including nested/shared/repeated-row alias behavior.
Three interleaved rounds, timing and tracemalloc peaks measured separately.10k x20 numeric table peak1790600 ->171848bytes (90.4% reduction);10k x22 mixed temporal peak11832067 ->9957363bytes (15.8% reduction). Wide numeric median time36.07 ->37.63ms; mixed165.86 ->161.13ms. Memory win established, no stable timing improvement claimed. Peaks are not cumulative allocations.
Five focused tests pass; full suite594pass/191fail on both variants with identical failure sets. Existing fixture drift tracked by #80. Runner and raw samples committed at packages/flint-py/tools/benchmark_temporal_columns.py and benchmark_temporal_columns_results.json. Extract baseline resolve_semantics.py with git show a794557:packages/flint-py/flint/core/resolve_semantics.py, then pass that path with --baseline-ref a794557 --rounds3 as documented by the runner.