Repository navigation
Memory usage increase after 3.0.61 #2219
Description
observations
Our ci/cd runners are failing due to a OOM issue after trying to upgrade to 3.0.61
When running 3.0.61 our runners pass
suspect code
I suspect it's due to the introduction of the
getContents() for reading external markdown files into table descriptions
#2164
suggested change
- make getContents and OPTIONAL feature activated using a flag / setting
AI explainer of possible issue
How getContents() works in PR #2164
Synchronous I/O: The helper uses a synchronous file system read—typically calling Node's fs.readFileSync(path, 'utf8') inside the Dataform context.
Execution Context: The file reading occurs inside the JavaScript compilation sandbox during VM execution. When you run dataform compile, Dataform spins up a virtual machine runner (under the hood, it transitioned through vm2 updates to isolated execution frames) to evaluate your project's JS and .sqlx configurations.
Object Retainers: The retrieved markdown string is bound to the table, view, or column's description metadata fields. This means the entire string content is held in active memory inside the generated compiled graph (CompiledGraph protocol buffer/JSON AST structure) until the compilation process completely serializes and exits.
update:
we increased the memory of our runners for the time being
Activity
ihistand commentedon Jul 15, 2026
Had a look at the code on main to understand the mechanism — a couple of things that might help narrow this down.
Small correction on the mechanism: getContents() doesn't fs.readFileSync the file directly. It goes through nativeRequire() (core/session.ts), so the .md is run through the module compiler (which turns it into exports.contents = \…
) and cached in the **module/requirecache**. The end-state retention conclusion is similar to the AI explainer, but via the require cache rather than a raw sync read.The read is already lazy / opt-in, which is the important part. In dataformCompile (core/main.ts), the loop that eagerly requires definition files filters to .js/.sqlx only:
So .md files are never loaded just by existing in the project — a markdown file's contents only enter memory when a definition explicitly calls getContents('./x.md'), and repeat calls on the same file are de-duped by the require cache. In other words, the flag you're suggesting is effectively already the behavior: no getContents() call ⇒ zero cost from this feature.
That makes me think the OOM correlates with actual getContents() usage (or possibly an unrelated change in 3.0.61), rather than the upgrade alone. Worth checking:
-- Do your compiled projects call getContents(), and if so, on how many / how large markdown files?
-- Does a compile of the same project pinned to 3.0.60 vs 3.0.61 differ in peak memory? (/usr/bin/time -v dataform compile or --max-old-space-size + a heap snapshot would isolate it.)
On the retention that does occur: each referenced markdown string is held for the duration of a single compile process (which is short-lived and frees everything on exit) — retained because both the require cache and the description field in the CompiledGraph reference it. It's bounded by the total size of markdown you actually pull in via getContents, so it'd only approach an OOM with very large or very many referenced docs on a tightly-capped runner. If that's the situation, a size cap in getContents (erroring on unreasonably large files) or streaming would be a more targeted fix than a global flag — but confirming it tracks getContents usage first would tell you whether that's even the right lever.
kolina commentedon Jul 25, 2026
I also don't think that
getContentscould be the reason unless you started to use it.We upgraded a couple of dependencies in
3.0.61, this could be the reason for increased memory usage.@frans-snyders, will you be able to provide a minimum example to reproduce it? Alternatively you can run bisect on commits in
3.0.61to find the culprit.