Skip to content

test(lambda): restore RIC invoke-lifecycle coverage in CI - #6829

Closed
Eren-Jeager123 wants to merge 49 commits into
mainfrom
test/lambda-ric-invoke-coverage
Closed

Eren-Jeager123 wants to merge 49 commits into
mainfrom
test/lambda-ric-invoke-coverage

Conversation

@Eren-Jeager123

@Eren-Jeager123 Eren-Jeager123 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem

No Lambda test exercises the invoke path. Every suite runs as docker run --entrypoint python -m pytest, which replaces ENTRYPOINT ["/lambda_entrypoint.sh", "python", "-m", "awslambdaric"] — so awslambdaric never boots. We cover imports, CUDA, GPU inference and S3 streaming, but not the RIC. The baked serving handlers have no coverage, and test_lambda_runtime.py asserts only that register_pre_fork is callable, not that it runs.

#6809 removed the tests that did cover this. Both were built around AWS_LAMBDA_CONCURRENCY_MODE and gated on the preview configs, so both would have gone inert anyway once the preview images were dropped — CI would have reported green while running nothing.

What this adds

Two tiers, because they fail in different ways.

1. Emulator (ric_concurrency_check.sh) runs the real chain — entrypoint shim → aws-lambda-rie → python -m awslambdaric → handler — against the two shapes awslambdaric 4.1.0 has:

Shape Env Expected
multi AWS_LAMBDA_MAX_CONCURRENCY=N exactly N forked workers, pre-fork hooks run
ondemand unset one worker, no pre-fork hooks

"Exactly N" comes from the RIC: run_concurrent() runs _before_fork (importing the handler and calling get_pre_fork() hooks in the parent), then forks exactly max_concurrency multiprocessing.Process workers.

2. Real Lambda (platform/run_platform.py) creates a function from the image under test, on managed instances with a GPU, and invokes it. This reaches what the emulator cannot: the real control plane, real instance provisioning, and the concurrency shape as configured through the API rather than an env var.

Asserted on both tiers: echo round-trip; awslambdaric/boto3/engine imports from handler code reached via an invoke; N concurrent invokes on N distinct workers; register_pre_fork ran in the parent; and for engines, N concurrent real inferences against exactly one shared GPU process.

The hook check is stronger than file existence — the hook writes its own PID and the worker asserts marker_pid == os.getppid(), proving it ran in the parent before the fork.

ric_probe_handler.py is shared by both tiers and is deliberately not named test_*.py so pytest does not collect it. platform/Dockerfile.test bakes it into the image, because Lambda cannot bind-mount a handler.

Validation

GPU paths run in this PR's validation-test. RIC mechanics were verified locally first on a CPU image built to the same entrypoint contract, with real aws-lambda-rie and real awslambdaric==4.1.0 — 9/9 checks passed. Negative control: stripping the @register_pre_fork block makes the hook check fail and the script exit 1, so the assertion is not vacuous.

The real-Lambda driver was run end to end against an actual function on managed-instance GPU capacity before this PR: function created, published, environment provisioned, then echo round-trip, imports through the invoke path, 2 concurrent invokes on 2 distinct workers, and the pre-fork hook confirmed in the parent.

That run established several API details worth recording, since each one failed first:

  • ExecutionEnvironmentConcurrencyConfig is rejected for managed-instance functions. Per-environment concurrency comes from PerExecutionEnvironmentMaxConcurrency inside CapacityProviderConfig, and the platform injects AWS_LAMBDA_MAX_CONCURRENCY from it. There is no ConcurrencyMode knob on this compute type.
  • GpuConfig is no longer accepted; the field is AcceleratorConfig, and AcceleratorMemorySize must be one of 3, 6, 12, 16, 24 or 48 GB.
  • A GPU capacity provider rejects a function that declares no accelerator requirement.
  • A new function settles into ActiveNonInvocable. It becomes invocable only after PublishVersion with PublishTo=LATEST_PUBLISHED, publishing is rejected until LastUpdateStatus reports Successful, and invokes then target the $LATEST.PUBLISHED qualifier.
  • Omitting PerExecutionEnvironmentMaxConcurrency does not pin one worker per environment — 2 concurrent invokes were still served by 2 workers. The worker-count assertion therefore applies to the multi shape only, which is the driver's default.

Notes for review

  • CI time: engine configs pay two engine cold starts on the emulator tier, one per shape. The real-Lambda tier adds one function lifecycle, mostly instance provisioning (~3-4 min observed).
  • Capacity: no GPU instance available exits 2, which the job turns into a warning rather than a release-blocking failure. A genuine assertion failure exits 1 and blocks.
  • The managed-instance API fields are absent from the bundled botocore, so the driver signs its own requests with SigV4 rather than shipping a service model. Post-GA that collapses to boto3.client("lambda").
  • The RIE returns HTTP 200 even when the handler raises (error in the body), so every emulator check validates a real JSON field rather than the status code. Worth preserving.
  • DOCKER_GPU_FLAG can be set empty to run the emulator check on a CPU box.
  • The real-Lambda job is wired in the PR workflows under the same condition as the GPU validation tests, so unrelated changes do not consume GPU capacity.
  • Drive-by: runai_s3_check.sh referenced the deleted rie_topology_check.sh.

sirutBuasai and others added 30 commits June 5, 2026 10:52
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* add lambda ric patch

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* run single line copy

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* deploy gamma with new RIC

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* revert to prod deployment

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

---------

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
# Conflicts:
#	.github/workflows/dispatch-release-lambda.yml

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* upgrade lambda ric version

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* fix paths

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* add additional tests

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* fix

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* update pytorch

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

---------

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* re add sglang

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* add vllm

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* add vllm cache

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* add timeout

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* fix telemetry

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

---------

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
sirutBuasai and others added 17 commits August 17, 2026 10:23
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* dockerfmt

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* add lambda ric test

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* fix json jq

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* use jq

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* add warmup

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* fix vllm test

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* fix fasterlock

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

---------

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* update sglang and vllm

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* add allowlist

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

---------

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* add s3 streaming for lambda inference engine

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* wire s3 test

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* forward credentials

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* server without rie

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

* fix curl

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>

---------

Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
…add modelscope (#6764)

* fix(lambda): report the real framework version for pytorch and vllm images

pytorch configs claimed 2.11.0 while pyproject pins torch==2.13.0, and the
vllm configs carried the CUDA version (13.0.3) instead of vllm 0.27.1. The
value is baked into bash_telemetry.sh at build time, so usage metrics for
these four images were labelled with the wrong version.

Release tags are unaffected: the lambda entry in DLContainersReleaseLogicPython
frameworks.yml renders release_tag as "{container_type}" and never references
{version}, and its version_pattern is '.*'.

Signed-off-by: Kevin Wang <kwanggg@amazon.com>

* fix(lambda): bump transformers to 5.10.4 for CVE-2026-9856

Path traversal via save_pretrained() in transformers <=5.8.0.dev0, fixed
in 5.10.0. 5.10.0 itself is yanked upstream, so pin 5.10.4.

* fix(lambda): bump vllm to 0.29.0 for CVE-2026-90553

RCE in the LlavaOnevision2 processor loader (ignores trust_remote_code),
fixed in 0.28.0; take 0.29.0 as current. flashinfer moves to 0.6.18 to
match 0.29.0's requirements/cuda.txt. The torch trio pin is unchanged
(2.13.0 / 2.11.0 / 0.28.0), so torch-constraints.txt still applies, and
the build-system requires are identical between the two refs.

* feat(lambda): bump vllm to 0.30.0, align transformers, add modelscope

vllm 0.30.0 (ref ced6857a) + flashinfer 0.6.18.post1 per its cuda.txt.

vllm asks for an unbounded transformers>=5.10.4 while its upstream test
suite hard-pins transformers==5.16.1, so the test env was swapping out
the transformers the image ships. Pin 5.16.1 in the pytorch pyproject
(the base every engine builder inherits) and add it to the vllm
constraint file so the runtime resolve can't drift. safetensors moves
0.8.0rc1 -> 0.8.0 because transformers 5.16.1 needs the final release.

modelscope<1.38 gives the vllm image ModelScope hub loading, matching
the al2023 image's serving extras; sglang already gets it via
sglang[all]. The cap is vllm#47325.

The spec-decode examples now pass --gpu-memory-utilization 0.95: at the
0.9 default vllm 0.29+ leaves ~0.09 GiB for KV on the 1xL4 runner, which
OOM'd the eagle case at max-model-len 2048.

* fix(lambda): align vllm ref and transformers with main's AL2023 image

Use the same vllm_ref as main's AL2023 vllm config (ec4a3a53, upstream's
'[CI] Bump Transformers version to 5.17.0') instead of the v0.30.0 tag,
so the upstream test suite checks out the ref the image is built from and
its transformers==5.17.0 test pin matches what the image ships.

Revert the hand-edit of the shared vllm_ec2_examples_test.sh and take
main's copy, which already carries the gpu-memory-utilization fix.

* fix(lambda): drop redundant transformers constraint

The pytorch pyproject pin already fixes the version the vllm image ships:
the vllm builder inherits /var/lang from the pytorch builder and uv leaves
an already-satisfied requirement alone.

* fix(lambda): install DeepGEMM/DeepSelect build deps, floor anyio for CVE-2026-63374

The vllm wheel build failed compiling deepgemm_C: DeepGEMM's vendored
deep_jit needs elfutils/libdwfl.h. Install elfutils-devel and gcc14 and
build with gcc14, matching docker/vllm/Dockerfile.amzn2023, which builds
the same vllm ref green.

anyio 4.13.0 (transitive via httpx) is CVE-2026-63374 CRITICAL, failing
the pytorch and sglang scans; floor it to 4.14.2.

* fix(lambda): drop added comments

---------

Signed-off-by: Kevin Wang <kwanggg@amazon.com>
…mages (#6791)

* fix(lambda): point runtime caches at /tmp in base, cupy and pytorch images

Lambda mounts everything except /tmp read-only, but only the sglang and
vllm stages redirected their caches there. CuPy JIT-compiles kernels and
writes them to $HOME/.cupy by default, so a customer's first custom
kernel fails on a read-only HOME; numba, huggingface_hub, triton and
inductor have the same problem in the cupy and pytorch images.

Sets HOME, USER and the XDG dirs on all three stages, CUPY_CACHE_DIR and
NUMBA_CACHE_DIR on cupy, and HF_HOME, TRITON_CACHE_DIR and
TORCHINDUCTOR_CACHE_DIR on pytorch, matching what the engine stages
already do. Preview stages inherit via FROM.

Adds env assertions plus a cupy GPU test that compiles a RawKernel and
checks the cache landed under /tmp — the case that was reported.

* fix(lambda): source telemetry from $HOME/.bashrc in base, cupy and pytorch

Setting HOME=/tmp moved the interactive-shell rc file off /root/.bashrc,
which is where the telemetry hook was written — amzn2023 bash only reads
/etc/bashrc via ~/.bashrc, so the hook stopped firing and four telemetry
tests failed. Write the hook to $HOME/.bashrc too, as the sglang and vllm
stages already do.
The cupy image advertises Numba, but numba.cuda JIT-compiles kernels through
libNVVM, which the CUDA runtime base omits: cuda.is_available() returned False
and every @cuda.jit raised NvvmSupportError. CuPy is unaffected because it
compiles via NVRTC, which the runtime base does ship.

Copies nvvm/lib64 and nvvm/libdevice from the devel base and points CUDA_HOME
at them. nvvm/bin (cicc, 73 MB) is left out — numba loads libnvvm.so directly
and never invokes the nvcc driver, so the image grows by ~62 MB rather than 134.

Covered by a GPU-free compile_ptx unit test and a @cuda.jit launch test, both
verified to fail against an image without libNVVM.
…6809)

register_pre_fork ships in the public RIC as of 4.1.0, so the separate
preview images exist for nothing. Pin awslambdaric==4.1.0 in the base,
cupy and pytorch closures and install it from PyPI — 4.1.0 publishes
manylinux/musllinux wheels for cp310-cp315 with a prebuilt
runtime_client.so, so the RIC builder stage and its gcc/cmake/autotools
install go away along with the S3 tarball fetch in pre_build.sh.

Removes the 5 *-preview configs, the builder-ric-preview stage, the 5
*-preview-py3 targets and ARG AWSLAMBDARIC_VERSION.

The GA RIC has no AWS_LAMBDA_CONCURRENCY_MODE: multi-concurrency is
entered by AWS_LAMBDA_MAX_CONCURRENCY alone and workers are forked
processes, so thread and hybrid go with the preview images. That drops
the run-ric-topology-test input, the lambdaric-topology-test job, the RIE
topology + mode-override validation steps, rie_topology_check.sh and
test/lambda/platform/ (run_topology.py, test_handler.py, Dockerfile.test).

Both serving handlers lose the try/except ImportError and mode check:
register_pre_fork when AWS_LAMBDA_MAX_CONCURRENCY is set, module-level
otherwise. test_lambda_runtime.py drops its concurrency-mode tests and
asserts register_pre_fork is importable in every image instead.
Every Lambda test runs as `docker run --entrypoint python -m pytest`, which
replaces the image ENTRYPOINT, so awslambdaric never boots and nothing exercises
the invoke path. #6809 removed the two tests that did (rie_topology_check.sh and
test/lambda/platform/): both were built around AWS_LAMBDA_CONCURRENCY_MODE and
both were gated on the preview configs, so both would have gone inert anyway when
the preview images were dropped.

ric_concurrency_check.sh runs the real chain — entrypoint shim -> aws-lambda-rie
-> python -m awslambdaric -> handler — and asserts the two shapes awslambdaric
4.1.0 actually has:

  multi     AWS_LAMBDA_MAX_CONCURRENCY=N -> exactly N forked workers, hooks run
  ondemand  unset                        -> one worker, no hooks

Exactly N comes from the RIC itself: run_concurrent() forks max_concurrency
multiprocessing.Process workers after running the pre-fork hooks in the parent.

Covers echo round-trip, library imports reached through the RIC, N concurrent
invokes landing on N distinct workers, register_pre_fork running in the parent,
and for engines N real inferences against one shared GPU process.

The hook assertion compares the marker PID against the worker's getppid(), so it
proves the hook ran in the parent before the fork rather than just that a file
exists. ric_probe_handler.py is named so pytest does not collect it.

Runs for all 5 configs in validation-test, not only previews.

Signed-off-by: Kevin Wang <kwanggg@amazon.com>
The try/except around the lambda_concurrency_hooks import let the probe degrade
silently on a pre-4.1.0 RIC, reporting a failed hook assertion instead of the real
cause. 4.1.0 is pinned in every image closure, so an older RIC is a defect to fail
on, not to tolerate — #6809 removed the same guard from the serving handlers.

Importing register_pre_fork unconditionally now surfaces a pre-4.1.0 RIC as a
Runtime.ImportModuleError at init. The baked-handler import is gated on the file
existing instead of a broad except, so core images (which ship no handler.py) are
still handled while an engine handler that fails to import now propagates.

wait_ready greps the container logs for the init error before tailing, since RIE
chatter otherwise pushes it out of the tail window.

Verified on a scratch image pinned to awslambdaric 4.0.0: both shapes fail, exit 1,
and the output names the cause —
  Unable to import module 'ric_probe_handler': No module named
  'awslambdaric.lambda_concurrency_hooks'
4.1.0 image still passes 9/9.

Signed-off-by: Kevin Wang <kwanggg@amazon.com>
The RIE check covers the runtime client against a local emulator. This adds the tier the
emulator cannot reach: a real Lambda function, built from the image under test, running on
managed instances with a GPU.

run_platform.py creates the function, publishes it, waits for an execution environment,
then asserts the invoke path end to end: an echo round-trip through the RIC, DLC libraries
importable from handler code reached by an invoke, N concurrent invokes served by N distinct
forked workers, and the register_pre_fork hook having run once in the parent. For engine
images it also drives N concurrent real inferences and asserts exactly one process holds the
GPU. Exit 2 means the capacity pool had no GPU instance, which is reported as a warning
rather than a failure.

Dockerfile.test bakes the probe handler into the image under test, because Lambda cannot
bind-mount a handler the way the container tests do.

The managed-instance fields are absent from the bundled botocore, so requests are signed
with SigV4 and sent directly. That keeps the test free of a service model and an S3 fetch.
POST-GA this collapses to boto3.client("lambda").

Verified against real Lambda in a development account, which established several contract
details that are not in the public API surface:

- ExecutionEnvironmentConcurrencyConfig is rejected for managed-instance functions. Per
  environment concurrency comes from PerExecutionEnvironmentMaxConcurrency inside
  CapacityProviderConfig, and the platform injects AWS_LAMBDA_MAX_CONCURRENCY from it.
- GpuConfig is no longer accepted. The field is AcceleratorConfig, and AcceleratorMemorySize
  must be one of 3, 6, 12, 16, 24 or 48 GB.
- A GPU capacity provider rejects a function that declares no accelerator requirement.
- A new function settles into ActiveNonInvocable. It becomes invocable only after
  PublishVersion with PublishTo=LATEST_PUBLISHED, and publishing is rejected until
  LastUpdateStatus reports Successful. Invokes target the $LATEST.PUBLISHED qualifier.
- Omitting PerExecutionEnvironmentMaxConcurrency does not pin one worker per environment, so
  the worker count assertion applies to the multi shape only and the default is that shape.

The job runs on the x86 build fleet because it builds and pushes the overlay image, gates
release alongside the other tests, and is wired in the PR workflows under the same condition
as the GPU validation tests so unrelated changes do not consume GPU capacity.
@Eren-Jeager123
Eren-Jeager123 marked this pull request as draft October 6, 2026 20:07
@Eren-Jeager123

Copy link
Copy Markdown
Contributor Author

DONT MERGE

Base automatically changed from lambda to main October 7, 2026 05:45
@Eren-Jeager123
Eren-Jeager123 deleted the test/lambda-ric-invoke-coverage branch October 7, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants