Repository navigation
test(lambda): restore RIC invoke-lifecycle coverage in CI - #6829
Closed
Eren-Jeager123 wants to merge 49 commits into
Closed
Eren-Jeager123 wants to merge 49 commits into
Eren-Jeager123 wants to merge 49 commits into
Conversation
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* add lambda ric patch Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * run single line copy Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * deploy gamma with new RIC Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * revert to prod deployment Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> --------- Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
# Conflicts: # .github/workflows/dispatch-release-lambda.yml Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* upgrade lambda ric version Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix paths Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add additional tests Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * update pytorch Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> --------- Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* re add sglang Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add vllm Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add vllm cache Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add timeout Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix telemetry Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> --------- Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* dockerfmt Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add lambda ric test Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix json jq Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * use jq Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add warmup Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix vllm test Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix fasterlock Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> --------- Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* update sglang and vllm Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * add allowlist Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> --------- Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
* add s3 streaming for lambda inference engine Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * wire s3 test Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * forward credentials Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * server without rie Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> * fix curl Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com> --------- Signed-off-by: sirutBuasai <sirutbuasai27@outlook.com>
…add modelscope (#6764) * fix(lambda): report the real framework version for pytorch and vllm images pytorch configs claimed 2.11.0 while pyproject pins torch==2.13.0, and the vllm configs carried the CUDA version (13.0.3) instead of vllm 0.27.1. The value is baked into bash_telemetry.sh at build time, so usage metrics for these four images were labelled with the wrong version. Release tags are unaffected: the lambda entry in DLContainersReleaseLogicPython frameworks.yml renders release_tag as "{container_type}" and never references {version}, and its version_pattern is '.*'. Signed-off-by: Kevin Wang <kwanggg@amazon.com> * fix(lambda): bump transformers to 5.10.4 for CVE-2026-9856 Path traversal via save_pretrained() in transformers <=5.8.0.dev0, fixed in 5.10.0. 5.10.0 itself is yanked upstream, so pin 5.10.4. * fix(lambda): bump vllm to 0.29.0 for CVE-2026-90553 RCE in the LlavaOnevision2 processor loader (ignores trust_remote_code), fixed in 0.28.0; take 0.29.0 as current. flashinfer moves to 0.6.18 to match 0.29.0's requirements/cuda.txt. The torch trio pin is unchanged (2.13.0 / 2.11.0 / 0.28.0), so torch-constraints.txt still applies, and the build-system requires are identical between the two refs. * feat(lambda): bump vllm to 0.30.0, align transformers, add modelscope vllm 0.30.0 (ref ced6857a) + flashinfer 0.6.18.post1 per its cuda.txt. vllm asks for an unbounded transformers>=5.10.4 while its upstream test suite hard-pins transformers==5.16.1, so the test env was swapping out the transformers the image ships. Pin 5.16.1 in the pytorch pyproject (the base every engine builder inherits) and add it to the vllm constraint file so the runtime resolve can't drift. safetensors moves 0.8.0rc1 -> 0.8.0 because transformers 5.16.1 needs the final release. modelscope<1.38 gives the vllm image ModelScope hub loading, matching the al2023 image's serving extras; sglang already gets it via sglang[all]. The cap is vllm#47325. The spec-decode examples now pass --gpu-memory-utilization 0.95: at the 0.9 default vllm 0.29+ leaves ~0.09 GiB for KV on the 1xL4 runner, which OOM'd the eagle case at max-model-len 2048. * fix(lambda): align vllm ref and transformers with main's AL2023 image Use the same vllm_ref as main's AL2023 vllm config (ec4a3a53, upstream's '[CI] Bump Transformers version to 5.17.0') instead of the v0.30.0 tag, so the upstream test suite checks out the ref the image is built from and its transformers==5.17.0 test pin matches what the image ships. Revert the hand-edit of the shared vllm_ec2_examples_test.sh and take main's copy, which already carries the gpu-memory-utilization fix. * fix(lambda): drop redundant transformers constraint The pytorch pyproject pin already fixes the version the vllm image ships: the vllm builder inherits /var/lang from the pytorch builder and uv leaves an already-satisfied requirement alone. * fix(lambda): install DeepGEMM/DeepSelect build deps, floor anyio for CVE-2026-63374 The vllm wheel build failed compiling deepgemm_C: DeepGEMM's vendored deep_jit needs elfutils/libdwfl.h. Install elfutils-devel and gcc14 and build with gcc14, matching docker/vllm/Dockerfile.amzn2023, which builds the same vllm ref green. anyio 4.13.0 (transitive via httpx) is CVE-2026-63374 CRITICAL, failing the pytorch and sglang scans; floor it to 4.14.2. * fix(lambda): drop added comments --------- Signed-off-by: Kevin Wang <kwanggg@amazon.com>
…mages (#6791) * fix(lambda): point runtime caches at /tmp in base, cupy and pytorch images Lambda mounts everything except /tmp read-only, but only the sglang and vllm stages redirected their caches there. CuPy JIT-compiles kernels and writes them to $HOME/.cupy by default, so a customer's first custom kernel fails on a read-only HOME; numba, huggingface_hub, triton and inductor have the same problem in the cupy and pytorch images. Sets HOME, USER and the XDG dirs on all three stages, CUPY_CACHE_DIR and NUMBA_CACHE_DIR on cupy, and HF_HOME, TRITON_CACHE_DIR and TORCHINDUCTOR_CACHE_DIR on pytorch, matching what the engine stages already do. Preview stages inherit via FROM. Adds env assertions plus a cupy GPU test that compiles a RawKernel and checks the cache landed under /tmp — the case that was reported. * fix(lambda): source telemetry from $HOME/.bashrc in base, cupy and pytorch Setting HOME=/tmp moved the interactive-shell rc file off /root/.bashrc, which is where the telemetry hook was written — amzn2023 bash only reads /etc/bashrc via ~/.bashrc, so the hook stopped firing and four telemetry tests failed. Write the hook to $HOME/.bashrc too, as the sglang and vllm stages already do.
The cupy image advertises Numba, but numba.cuda JIT-compiles kernels through libNVVM, which the CUDA runtime base omits: cuda.is_available() returned False and every @cuda.jit raised NvvmSupportError. CuPy is unaffected because it compiles via NVRTC, which the runtime base does ship. Copies nvvm/lib64 and nvvm/libdevice from the devel base and points CUDA_HOME at them. nvvm/bin (cicc, 73 MB) is left out — numba loads libnvvm.so directly and never invokes the nvcc driver, so the image grows by ~62 MB rather than 134. Covered by a GPU-free compile_ptx unit test and a @cuda.jit launch test, both verified to fail against an image without libNVVM.
…6809) register_pre_fork ships in the public RIC as of 4.1.0, so the separate preview images exist for nothing. Pin awslambdaric==4.1.0 in the base, cupy and pytorch closures and install it from PyPI — 4.1.0 publishes manylinux/musllinux wheels for cp310-cp315 with a prebuilt runtime_client.so, so the RIC builder stage and its gcc/cmake/autotools install go away along with the S3 tarball fetch in pre_build.sh. Removes the 5 *-preview configs, the builder-ric-preview stage, the 5 *-preview-py3 targets and ARG AWSLAMBDARIC_VERSION. The GA RIC has no AWS_LAMBDA_CONCURRENCY_MODE: multi-concurrency is entered by AWS_LAMBDA_MAX_CONCURRENCY alone and workers are forked processes, so thread and hybrid go with the preview images. That drops the run-ric-topology-test input, the lambdaric-topology-test job, the RIE topology + mode-override validation steps, rie_topology_check.sh and test/lambda/platform/ (run_topology.py, test_handler.py, Dockerfile.test). Both serving handlers lose the try/except ImportError and mode check: register_pre_fork when AWS_LAMBDA_MAX_CONCURRENCY is set, module-level otherwise. test_lambda_runtime.py drops its concurrency-mode tests and asserts register_pre_fork is importable in every image instead.
Every Lambda test runs as `docker run --entrypoint python -m pytest`, which replaces the image ENTRYPOINT, so awslambdaric never boots and nothing exercises the invoke path. #6809 removed the two tests that did (rie_topology_check.sh and test/lambda/platform/): both were built around AWS_LAMBDA_CONCURRENCY_MODE and both were gated on the preview configs, so both would have gone inert anyway when the preview images were dropped. ric_concurrency_check.sh runs the real chain — entrypoint shim -> aws-lambda-rie -> python -m awslambdaric -> handler — and asserts the two shapes awslambdaric 4.1.0 actually has: multi AWS_LAMBDA_MAX_CONCURRENCY=N -> exactly N forked workers, hooks run ondemand unset -> one worker, no hooks Exactly N comes from the RIC itself: run_concurrent() forks max_concurrency multiprocessing.Process workers after running the pre-fork hooks in the parent. Covers echo round-trip, library imports reached through the RIC, N concurrent invokes landing on N distinct workers, register_pre_fork running in the parent, and for engines N real inferences against one shared GPU process. The hook assertion compares the marker PID against the worker's getppid(), so it proves the hook ran in the parent before the fork rather than just that a file exists. ric_probe_handler.py is named so pytest does not collect it. Runs for all 5 configs in validation-test, not only previews. Signed-off-by: Kevin Wang <kwanggg@amazon.com>
The try/except around the lambda_concurrency_hooks import let the probe degrade silently on a pre-4.1.0 RIC, reporting a failed hook assertion instead of the real cause. 4.1.0 is pinned in every image closure, so an older RIC is a defect to fail on, not to tolerate — #6809 removed the same guard from the serving handlers. Importing register_pre_fork unconditionally now surfaces a pre-4.1.0 RIC as a Runtime.ImportModuleError at init. The baked-handler import is gated on the file existing instead of a broad except, so core images (which ship no handler.py) are still handled while an engine handler that fails to import now propagates. wait_ready greps the container logs for the init error before tailing, since RIE chatter otherwise pushes it out of the tail window. Verified on a scratch image pinned to awslambdaric 4.0.0: both shapes fail, exit 1, and the output names the cause — Unable to import module 'ric_probe_handler': No module named 'awslambdaric.lambda_concurrency_hooks' 4.1.0 image still passes 9/9. Signed-off-by: Kevin Wang <kwanggg@amazon.com>
The RIE check covers the runtime client against a local emulator. This adds the tier the
emulator cannot reach: a real Lambda function, built from the image under test, running on
managed instances with a GPU.
run_platform.py creates the function, publishes it, waits for an execution environment,
then asserts the invoke path end to end: an echo round-trip through the RIC, DLC libraries
importable from handler code reached by an invoke, N concurrent invokes served by N distinct
forked workers, and the register_pre_fork hook having run once in the parent. For engine
images it also drives N concurrent real inferences and asserts exactly one process holds the
GPU. Exit 2 means the capacity pool had no GPU instance, which is reported as a warning
rather than a failure.
Dockerfile.test bakes the probe handler into the image under test, because Lambda cannot
bind-mount a handler the way the container tests do.
The managed-instance fields are absent from the bundled botocore, so requests are signed
with SigV4 and sent directly. That keeps the test free of a service model and an S3 fetch.
POST-GA this collapses to boto3.client("lambda").
Verified against real Lambda in a development account, which established several contract
details that are not in the public API surface:
- ExecutionEnvironmentConcurrencyConfig is rejected for managed-instance functions. Per
environment concurrency comes from PerExecutionEnvironmentMaxConcurrency inside
CapacityProviderConfig, and the platform injects AWS_LAMBDA_MAX_CONCURRENCY from it.
- GpuConfig is no longer accepted. The field is AcceleratorConfig, and AcceleratorMemorySize
must be one of 3, 6, 12, 16, 24 or 48 GB.
- A GPU capacity provider rejects a function that declares no accelerator requirement.
- A new function settles into ActiveNonInvocable. It becomes invocable only after
PublishVersion with PublishTo=LATEST_PUBLISHED, and publishing is rejected until
LastUpdateStatus reports Successful. Invokes target the $LATEST.PUBLISHED qualifier.
- Omitting PerExecutionEnvironmentMaxConcurrency does not pin one worker per environment, so
the worker count assertion applies to the multi shape only and the default is that shape.
The job runs on the x86 build fleet because it builds and pushes the overlay image, gates
release alongside the other tests, and is wired in the PR workflows under the same condition
as the GPU validation tests so unrelated changes do not consume GPU capacity.
Eren-Jeager123
marked this pull request as draft
October 6, 2026 20:07
Contributor
Author
|
DONT MERGE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
No Lambda test exercises the invoke path. Every suite runs as
docker run --entrypoint python -m pytest, which replacesENTRYPOINT ["/lambda_entrypoint.sh", "python", "-m", "awslambdaric"]— soawslambdaricnever boots. We cover imports, CUDA, GPU inference and S3 streaming, but not the RIC. The baked serving handlers have no coverage, andtest_lambda_runtime.pyasserts only thatregister_pre_forkiscallable, not that it runs.#6809 removed the tests that did cover this. Both were built around
AWS_LAMBDA_CONCURRENCY_MODEand gated on the preview configs, so both would have gone inert anyway once the preview images were dropped — CI would have reported green while running nothing.What this adds
Two tiers, because they fail in different ways.
1. Emulator (
ric_concurrency_check.sh) runs the real chain — entrypoint shim →aws-lambda-rie→python -m awslambdaric→ handler — against the two shapesawslambdaric4.1.0 has:multiAWS_LAMBDA_MAX_CONCURRENCY=Nondemand"Exactly N" comes from the RIC:
run_concurrent()runs_before_fork(importing the handler and callingget_pre_fork()hooks in the parent), then forks exactlymax_concurrencymultiprocessing.Processworkers.2. Real Lambda (
platform/run_platform.py) creates a function from the image under test, on managed instances with a GPU, and invokes it. This reaches what the emulator cannot: the real control plane, real instance provisioning, and the concurrency shape as configured through the API rather than an env var.Asserted on both tiers: echo round-trip;
awslambdaric/boto3/engine imports from handler code reached via an invoke; N concurrent invokes on N distinct workers;register_pre_forkran in the parent; and for engines, N concurrent real inferences against exactly one shared GPU process.The hook check is stronger than file existence — the hook writes its own PID and the worker asserts
marker_pid == os.getppid(), proving it ran in the parent before the fork.ric_probe_handler.pyis shared by both tiers and is deliberately not namedtest_*.pyso pytest does not collect it.platform/Dockerfile.testbakes it into the image, because Lambda cannot bind-mount a handler.Validation
GPU paths run in this PR's
validation-test. RIC mechanics were verified locally first on a CPU image built to the same entrypoint contract, with realaws-lambda-rieand realawslambdaric==4.1.0— 9/9 checks passed. Negative control: stripping the@register_pre_forkblock makes the hook check fail and the script exit 1, so the assertion is not vacuous.The real-Lambda driver was run end to end against an actual function on managed-instance GPU capacity before this PR: function created, published, environment provisioned, then echo round-trip, imports through the invoke path, 2 concurrent invokes on 2 distinct workers, and the pre-fork hook confirmed in the parent.
That run established several API details worth recording, since each one failed first:
ExecutionEnvironmentConcurrencyConfigis rejected for managed-instance functions. Per-environment concurrency comes fromPerExecutionEnvironmentMaxConcurrencyinsideCapacityProviderConfig, and the platform injectsAWS_LAMBDA_MAX_CONCURRENCYfrom it. There is noConcurrencyModeknob on this compute type.GpuConfigis no longer accepted; the field isAcceleratorConfig, andAcceleratorMemorySizemust be one of 3, 6, 12, 16, 24 or 48 GB.ActiveNonInvocable. It becomes invocable only afterPublishVersionwithPublishTo=LATEST_PUBLISHED, publishing is rejected untilLastUpdateStatusreportsSuccessful, and invokes then target the$LATEST.PUBLISHEDqualifier.PerExecutionEnvironmentMaxConcurrencydoes not pin one worker per environment — 2 concurrent invokes were still served by 2 workers. The worker-count assertion therefore applies to themultishape only, which is the driver's default.Notes for review
boto3.client("lambda").DOCKER_GPU_FLAGcan be set empty to run the emulator check on a CPU box.runai_s3_check.shreferenced the deletedrie_topology_check.sh.