Feature request: Skip `NodejsNpmEsbuildBuilder:NpmUpdate` step when using `--build-in-source` in a js/ts "monorepo"

[cdriesler]

Describe your idea/feature/enhancement

We are currently making great use of the new --build-in-source flag for sharing js/ts code between several AWS resources (also written in js/ts). Our project is set up in a "monorepo" structure, something like:

packages/
  package-a/
    package.json
endpoints/
  endpoint-a/
    package.json
package.json

We use npm workspaces in the root package.json to allow resources like endpoint-a to specify modules like package-a as a dependency. The built-in SAM esbuild options can then find and bundle the code when using --build-in-source.

We've noticed, though, that overall build times have increased by quite a bit because of (what seems to be) a redundant NodejsNpmEsbuildBuilder:NpmUpdate step for each resource. In our case, all of our required dependencies have already been installed before we run sam build. This is true for all resources, because they are all also specified in the workspace's package.json.

The second step of the build for each resource (NodejsNpmEsbuildBuilder:EsbuildBundle ) represents less than 5% of the build time. If we could intelligently or explicitly skip the NpmUpdate step, our stack builds would speed up considerably.

Proposal

• Option A: An explicit flag like --skip-dependencies to turn off SAM's management/installation of dependencies. Potentially also requiring --build-in-source. (It feels like that might be the only case where this new flag makes sense.)
• Option B: Skip redundant installs that can be asserted based on the contents of a root package.json. This seems a bit antithetical to the way SAM thinks about resources and handles building, but may be technically possible.

[cdriesler]

Somewhat related: this would also resolve our issue in #6491

[hnnasit]

Hi @cdriesler, thanks for the feature request. I will bring this up with the team and PM's.

[hnnasit]

We've noticed, though, that overall build times have increased by quite a bit because of (what seems to be) a redundant

This is interesting as npm caching should be able to reduce the build times. Would you be able to provide a sample project for us to investigate why build times are increasing?

[RobertMarton1985]

We are in the middle of updating our SAM project and we've ended up with the same project structure. We also use npm workspaces and we also have shared custom dependencies like package-a that is used by our lambda endpoints.
Using the --build-in-source flag is mandatory in this case otherwise package-a cannot be resolved when esbuild tries to create the bundle. It runs an npm install but since package-a is not on the registry, it fails.

Anyway, we have found that for each endpoint an npm update is executed with the --install-links flag which makes the build really slow. For example the esbuild bundle command is executed in 1-2 seconds, but the npm update takes more than a minute. And it is executed for each lambda function one-by-one.
We have checked the logs during the npm update and it looked like it was running on the ROOT project instead of the lambda. We guess this is because the workspaces do not have a package-lock.json file.

Note: We have noticed that the only way to get rid of the built in @aws-sdk v3 to be deployed with the lambda is to set the `External˙ build properties of the lambdas to an empty array []. With this setting we can achieve around 200ms cold start times. Every other settings resulted in much longer cold starts 600ms+. We strongly believe this is caused by the presence of the default aws sdk.
This also prevents us to use layers because those dependencies must be listed in the Externals property otherwise the esbuild failes. But if we add them as externals (eq. the Externals is not an empty array) then the default aws-sdk is also deployed. We definiately want to avoid that.

Note 2: We have a workaround but it is not ideal at all. We do not use the --build-in-source flag and we perform additional tasks via lerna on each lambda before sam build:

1. In each lambda package.json we have a script to bundle the code: esbuild ./ --bundle --platform=node --outfile=bundle.js --minify. We use this script to bundle the code.
2. After the bundle is created we remove the dependencies block completely from the package.json file.

During the sam build the esbuild is still performed but at that point the bundle is already created and it has no additional dependencies so it completes without error.

[adesso-os]

Another AWS product that doesn't understand that there are more package managers in the NodeJS space than npm, and then also doesn't provide any way to work around these annoying issues. Working with any software product out of AWS is such an endless pain.

The real workaround is to drop SAM ASAP and implement a real build.

[RobertMarton1985]

@hnnasit is there any update on this?

[ffraenz]

I’d like to revive this issue because I believe the current build behavior undermines security, reliability, and developer productivity.

Like others here, I’m working with a monorepo setup using NPM workspaces, where each Lambda function lives in its own package alongside shared library packages. I typically run sam build --build-in-source to build.

With this setup, I’m running into several recurring problems:

1. npm update vs. npm install vs. none of them

   Running npm update during builds is problematic. It introduces nondeterministic outcomes and unnecessary supply chain risk, especially in light of recent NPM security incidents. As a developer, I want to control when my dependencies update. During builds, installs should resolve consistently and deterministically. Ideally, builds should not trigger updates or reinstalls at all.

   Because SAM reinstalls or updates dependencies during each build, my editor often lags behind, flagging modules as missing until the process settles. This introduces avoidable bad DX.

2. Non-deterministic build failures

   My SAM builds frequently fail multiple times in a row before succeeding. A typical failure looks like this:

   Build Failed
   Error: NodejsNpmEsbuildBuilder:NpmUpdate - NPM Failed: npm warn Unknown cli config "--unsafe-perm". This will stop working in the next major version of npm.
   npm warn config production Use `--omit=dev` instead.
   npm warn workspaces @function/name in filter set, but no workspace folder present
   npm error code ENOTEMPTY
   npm error syscall rename
   npm error path /.../monorepo/node_modules/@aws-sdk/client-s3
   npm error dest /.../monorepo/node_modules/@aws-sdk/.client-s3-0OHxCkCC
   npm error errno -66
   npm error ENOTEMPTY: directory not empty, rename '/.../monorepo/node_modules/@aws-sdk/client-s3' -> '/.../monorepo/node_modules/@aws-sdk/.client-s3-0OHxCkCC'
   npm error A complete log of this run can be found in: ~/.npm/_logs/...
   

   The only workaround I’ve found is to run npm ci and then retry sam build --build-in-source multiple times until it eventually works. This is so tedious, sometimes I forget what I was doing in the first place.

If there's anything I can do to help make this happen, I'm here for it.

[RobertMarton1985]

@ffraenz By now we have a quite reliable and working process. I will try to outline our project structure here so you can adopt it if you want.

In our root package.json we define our workspaces like this:
"workspaces": [ "src/**/module" ]

We create the following structure for our lambdas and utility modules:

modules/
  module-1/
    module/
      package.json
      index.js
lambdas/
  lambda-1/
    module/
      package.json
      index.js
package.json

We also have our unit test scripts inside the module directories but that's not important right now.

In the package.json of every module we set the following parameters:

  "type": "module",
  "exports": "./index.js",

And we have a script in the package.json of the lambdas to bundle the code:

  "scripts": {
    "bundle": "esbuild ./index.js --bundle --format=cjs --main-fields=module,main --platform=node --outdir=bundle --minify",
    "bundle:dev": "esbuild ./index.js --bundle --format=cjs --main-fields=module,main --platform=node --outdir=bundle"
  }

This creates a bundle directory in each module and puts the bundled index.js file there.

We use lerna to bundle the modules.

lerna.json:

{
  "$schema": "node_modules/lerna/schemas/lerna-schema.json",
  "version": "0.0.0"
}

nx.json:

{
  "namedInputs": {
    "source": ["{projectRoot}/**/*.js", "!{projectRoot}/bundle/index.js"],
    "modules": ["{workspaceRoot}/src/modules/**/*.js", "!{workspaceRoot}/src/modules/**/tests/*.js"]
  },
  "targetDefaults": {
    "bundle": {
      "cache": true,
      "inputs": ["source", "modules"],
      "outputs": ["{projectRoot}/bundle/index.js"]
    },
    "bundle:dev": {
      "cache": true,
      "inputs": ["source", "modules"],
      "outputs": ["{projectRoot}/bundle/index.js"]
    }
  }
}

In the SAM templates we define the following properties:

  Lambda1Function
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Format: cjs
        Minify: false
        External: []
    Properties:
      CodeUri: ../../src/lambdas/lambda-1/module/bundle
      FunctionName: Lambda1

This setup uses the bundled index.js as the lambda code. The Metadata part is needed to ensure that the default aws-sdk is not added to the deployment package.
The CodeUri is relative so you need to set it according to the location of your sam template file.

We have scripts in our root package.json to bundle all lambdas:

    "bundle": "lerna run bundle",
    "bundle:dev": "lerna run bundle:dev",

After the bundle is ready we can simply run sam build and then sam deploy. We do NOT use --build-in-source

We have tried out lots of options and settled on this one eventually. We are using this for more than a year by now without any issue. Please let me know if you tried it or if you have any question. This is quite a complex stuff so maybe I missed out something.

[bnusunny]

Thanks for the detailed reports here — especially @RobertMarton1985 and @ffraenz, whose descriptions turned out to be exactly right about the mechanism. I reproduced this end to end and dug into where the time goes. Summary: this is reproducible on SAM CLI 1.166.2, and there are three distinct problems underneath the feature request, one of which is a correctness issue rather than a performance one.

Reproduction

npm workspaces monorepo, two esbuild functions, one shared local package (@mono/shared), all dependencies already installed at the root. SAM CLI 1.166.2, node 24.21.0, npm 11.19.0.

• sam build --build-in-source runs NodejsNpmEsbuildBuilder:NpmUpdate once per function, as reported.
• Without --build-in-source the build fails with npm error 404 Not Found - GET https://registry.npmjs.org/@mono%2fshared, so --build-in-source is indeed mandatory for this layout (confirming @RobertMarton1985).
• The install step emits npm warn workspaces @mono/endpoint-a in filter set, but no workspace folder present — npm resolves up to the workspace root, so the work is root-scoped, and it is repeated once per function. That is why npm's cache does not help: the cost is resolution/reconciliation of the whole monorepo tree, not package downloads.

Problem 1: the install step ignores your lockfile (correctness)

For build-in-source, get_install_action returns NodejsNpmUpdateAction, which runs:

npm update --no-audit --no-save --omit=dev --no-package-lock --install-links

--no-package-lock means the lockfile is not consulted. Measured on a fixture whose root package-lock.json pinned lodash@4.17.20 with a spec of ^4.17.20:

• before sam build: root node_modules/lodash = 4.17.20
• after sam build: root node_modules/lodash = 4.18.1

So a build silently upgrades dependencies in the developer's own source tree, despite a committed lockfile. This is @ffraenz's point, and it is a real defect: sam build is not supposed to be a dependency-update command. It is also the mechanism behind the intermittent ENOTEMPTY … rename failures — npm is renaming packages inside the shared root node_modules while an editor/tsserver (or another build) has the tree open — and behind #6491, since --omit=dev prunes devDependencies from the source tree.

Relevant code: aws_lambda_builders/workflows/nodejs_npm/workflow.py:264-265 and aws_lambda_builders/workflows/nodejs_npm/actions.py (NodejsNpmUpdateAction).

Problem 2: UseNpmCi: true cannot be reached here — and would be harmful if it were

get_install_action looks for the lockfile in the function's directory (workflow.py:247-248). With npm workspaces the lockfile only exists at the repo root, so UseNpmCi: true silently falls through to NpmUpdate. Verified: setting it in BuildProperties changes nothing, the log still shows NpmUpdate.

It is worth saying that simply making it reachable would be the wrong fix. Running npm ci --install-links from a workspace package operates on the root project and removes the root devDependencies — in my fixture it deleted node_modules/.bin/esbuild, i.e. the very binary the esbuild workflow resolves through npm root. So npm ci is not the escape hatch for this layout.

Problem 3: --cached cannot skip the install either

The mechanism that already exists for this (download_dependencies=False, used by the incremental build strategy) is unreachable under --build-in-source. The esbuild workflow returns early when building in source and never runs MoveDependenciesAction (aws_lambda_builders/workflows/nodejs_npm_esbuild/workflow.py:146-148), so .aws-sam/deps/<uuid> is never populated; IncrementalBuildStrategy then sees a missing dependencies directory on every run and sets download_dependencies=True again. Verified: three consecutive sam build --build-in-source --cached runs each ran NpmUpdate, and .aws-sam/deps/ stayed empty. sam sync reads the same flag.

What the numbers say about the fix

Each command below was run in the function directory of a pristine copy of the same fixture (root lockfile pinning lodash@4.17.20, esbuild as a root devDependency):

command	lockfile honored	root devDependencies	prunes a dep removed between builds (plain layout / workspaces)	wall time
npm update --no-audit --no-save --omit=dev --no-package-lock --install-links (current)	no — 4.17.20 → 4.18.1	kept	yes / no	582 ms
npm ci --install-links (UseNpmCi)	yes	esbuild removed	–	528 ms
npm install --no-audit --no-save --omit=dev --install-links	yes, no drift	kept	yes / no	290 ms

Two things follow. First, the motivation for #579 (npm install will not remove a dependency that was deleted from package.json between two builds) no longer holds on current npm: plain npm install prunes the removed package in the flat build-in-source layout, and in a workspaces monorepo neither command prunes it, so update buys nothing there while still bypassing the lockfile. Second, none of these commands modify or create a lockfile when --no-save is passed, which I checked separately.

Proposal

1. Use a lockfile-respecting install for build-in-source instead of npm update — drop --no-package-lock and use install. This makes builds deterministic, stops mutating the developer's dependency versions, keeps the pruning behaviour Fix stringifying λ environment variables when using Python2 #579 wanted, and is roughly 2x faster in a trivial fixture (much more in a real monorepo, since an up-to-date tree becomes a near no-op). I am preparing a PR against aws-lambda-builders for this.
2. Let --cached actually short-circuit the install under --build-in-source for the nodejs/esbuild workflows, so repeat builds skip dependency work entirely when the manifest has not changed. This is the fix that addresses the original request without any new user-facing surface.
3. Then consider the explicit opt-out from the description (Option A — e.g. a SkipDependencyInstall build property, or a CLI flag valid with --build-in-source), wired to the existing internal download_dependencies=False. This is still worth having for people who own their own build step, which is the ask in Feature request: Skip npm install #7847 — but after 1 and 2 it stops being the only way to get a reasonable build time.

Keeping this issue open as the tracking item for that work. #7847 is the same request from the "I already build my own bundles" angle and I'll close it as a duplicate of this one.