Skip to content

perf: add reproducible render and build baseline instrumentation #166

Closed

Description

@spboyer

Parent

#160

Validated infrastructure gap

At a794557384adcc3293a77043f6b9aff1c33d152c, package scripts and CI provide correctness/build checks but no dedicated repeatable performance baseline runner. MCP has in-process Vega-Lite, ECharts, and Chart.js renderers, so cold initialization, compilation, rendering, and artifact conversion need separate measurements rather than compiler-only timing.

A local npm run build:js on Node v26.12.1 failed with ERR_WORKER_OUT_OF_MEMORY during the tsup/DTS build under an approximately 2144 MiB V8 heap limit. This is diagnostic evidence only: CI uses Node 20/22, the host is shared, and there is no hosted-CI reproduction. The site CI job already explicitly allows a larger heap. Do not prescribe a memory-limit increase or claim a supported-runtime build defect without additional evidence.

Upstream issue listings and all 21 open PR titles were checked on 2026-11-11; no equivalent benchmark-infrastructure work was found.

Proposed scope

Add a small opt-in, reproducible benchmark runner using existing dependencies and deterministic fixtures. Start with compiler and supported in-process MCP render paths, cold and warm runs, and build command/environment reporting. Keep hosted-CI duration assessment separate and report unavailable evidence honestly.

Acceptance criteria

  • Record source revision, Node/platform identities, fixture shape/size, backend, output format/dimensions, cold vs warm state, individual samples, and summary timing plus memory metrics with precisely defined meanings.
  • Validate artifact output so errors/empty charts cannot appear as fast successful runs; preserve failed attempts and report skipped/unsupported cases explicitly.
  • Support at least three repeats; serialize benchmark runs on the shared host. Do not add a noisy performance gate to ordinary CI or change default npm build/test behavior.
  • Distinguish process RSS/heap snapshots from true allocation totals and instrumented timings from ordinary timings.
  • Document reproduction and limitations. Use existing tooling; avoid a large new benchmark framework.
  • Any later build or CI optimization must have its own material measured evidence and preserve coverage, packaging, and supported Node compatibility.
  • One focused instrumentation PR, no automatic merge. This issue is not itself a claim of runtime or CI speedup.

Activity

  1. Chenglong-MS commented on Nov 15, 2026

    @Chenglong-MS
    Contributor

    Done in #171 (merged into dev): npm run benchmark:render records cold and warm compiler and MCP render baselines; see docs/render-performance.md. Ships in 1.0.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions