[JunyiXu-nv]

@coderabbitai summary

Description

Stacked on #21393 (TRTLLM-17272 health-check + private-access lint, "[TRTLLM-17272][feat] Add public LLM health queries and a private-access lint for the serving frontends"): the first 2 commits below belong to that PR and disappear from this diff when it merges; this PR's own change is the last 2 commits.

The serving frontends (HTTP tensorrt_llm/serve, the CLI tensorrt_llm/commands, the OpenEngine gRPC frontend tensorrt_llm/grpc/openengine) still read private members of LLM, the executors, CompletionOutput, SamplingParams, MultimodalDataTracker, OpenAIDisaggServer and PyExecutor. This PR gives each remaining read a thin public entry point on the class that owns the member, switches the frontends to it with no behavior change, and shrinks the private-access-lint baseline from 50 recorded accesses to 19. It also adds tensorrt_llm/llmapi/engine_client.py: an EngineClient protocol naming the engine-level operations every frontend shares (generate, health, load), not enforced at runtime (the private-access lint is the enforcement) and deliberately without role-specific members such as encode_limits. After this PR, moving the HTTP handlers into serving classes carries only public calls.

Part of the trtllm-serve refactor: public runtime API for the frontends (epic TRTLLM-17056, task TRTLLM-17272). Builds on #21393.

Converted accesses (31 of 50; old read → new public member)

Where	Old read	New public member
serve/_frontend_processes.py _spawn_attached_frontends	llm._executor + isinstance + attach-info check	LLM.multi_frontend_attach_info() (same ValueError, byte-identical text)
serve/openai_server.py _record_generator_termination	getattr(generator, "_executor", None) isinstance proxy	LLM.engine_in_worker_process
serve/openai_server.py _init_llm	generator._hf_model_dir + tokenizer-name fallback	LLM.hf_model_dir (one fallback rule, see below)
serve/openai_server.py register_routes	generator._executor.resource_governor_queue	LLM.resource_governor_queue
serve/openai_server.py _init_embedding_batcher	generator._encoder_executor.model_engine limits	LLM.encode_limits (EncodeLimits dataclass; encode() reads the same definition)
serve/openai_server.py _on_launcher_lost	generator._executor + executor._set_fatal_error(...)	LLM.record_fatal_error() (+ existing LLM.shutdown_started); executor side GenerationExecutor.record_fatal_error()
serve/chat_utils.py (4 sites)	mm_data_tracker._multimodal_server_config	MultimodalDataTracker.multimodal_server_config
serve/openai_protocol.py	sampling_params._set_request_provided_fields(...)	SamplingParams.set_request_provided_fields(...) (public rename, same body)
serve/postprocess_handlers.py (4 sites)	getattr(output, '_spec_dec_counters', None)	CompletionOutput.spec_dec_counters (direct attribute; the test doubles gained the member)
serve/postprocess_handlers.py	output._last_text_len	CompletionOutput.text_diff_offset
commands/serve.py _bind_launcher_uds	llm._executor + getattr(..., "_multi_frontend_ipc_dir", None)	LLM.multi_frontend_ipc_dir (executor side: GenerationExecutor.multi_frontend_ipc_dir, proxy override)
commands/serve.py _run_fleet_worker_impl	server._config.hostname, server._config.port	OpenAIDisaggServer.address
grpc/openengine/capabilities.py	getattr(llm, "_hf_model_config", None)	LLM.hf_model_config (read stays a tolerant getattr for non-LLM doubles)
grpc/openengine/capabilities.py	getattr(type(input_processor), "_registered_model_type", None)	registered_model_type() in tensorrt_llm/inputs/registry.py, next to the decorator that stamps it
grpc/openengine/control.py _get_kv_cache_load	_executor + tolerant getter	LLM.get_kv_cache_load()
grpc/openengine/control.py GetServerInfo	_executor + tolerant getters	LLM.get_kv_cache_capacity() behind an args.encode_only guard; LLM.get_openengine_discovery()
grpc/openengine/control.py _requested_sources	_executor + tolerant getter	LLM.get_openengine_discovery()
grpc/openengine/formatting.py _token_holdback	getattr(sampling_params, "_stop_word_ids", None)	SamplingParams.stop_word_ids
grpc/openengine/node.py	executor._openengine_discovery = ..., executor._openengine_node_server = ... (writes)	PyExecutor.set_openengine_discovery(), PyExecutor.set_openengine_node_server()
grpc/openengine/request_mapping.py	sampling_params._set_request_provided_fields(...)	SamplingParams.set_request_provided_fields(...)
grpc/openengine/server.py OpenEngineServer.start	self._llm._executor.get_kv_cache_capacity	LLM.get_kv_cache_capacity()

Stays in the baseline, for other owners: _postprocess_result (9, owned by the post-processing strategy unification), ._done (5, owned by the public finished switch), the VisualGen output._save_bytes (1, public VisualGen output encoder follow-up), KVCacheEventManager._hash_cache_salt (1, needs a public nanobind binding from the KV-cache-manager owners), and OpenEngine's three reads of the private LlmArgs attributes _enable_routing_load / _openengine_discovery — those two are deliberately absent from the public LLM configuration and telemetry schema (llmapi/llm_args.py), so exposing them is an LLM-args change for its own PR; llmapi/llm_args.py is not edited here. Baseline: 50 → 19 accesses; MAX_TOTAL lowered to 19.

New public members (all status="prototype" where api-stability tracks them)

• LLM.multi_frontend_attach_info() -> dict — the attach payload for serving frontends sharing this LLM's engine; raises the exact startup ValueError otherwise.
• LLM.multi_frontend_ipc_dir -> Optional[str] — the launcher-owned multi-frontend IPC directory, or None.
• LLM.engine_in_worker_process -> bool — whether generation runs through the IPC executor proxy (engine in separate MPI worker processes).
• LLM.record_fatal_error(error) -> bool — records an unrecoverable engine error (first one wins); False without a generation executor. Executor side: GenerationExecutor.record_fatal_error().
• LLM.hf_model_dir -> Union[str, Path] — self._hf_model_dir or self.args.model, the rule _try_load_generation_config_explicit_values already used; now the only copy.
• LLM.hf_model_config -> Optional[transformers.PretrainedConfig] — the loaded HF config.
• LLM.resource_governor_queue -> Optional[Any] — the generation executor's queue; raises RuntimeError without a generation executor.
• LLM.encode_limits -> Optional[EncodeLimits] — the limits encode() checks (max_batch_size / max_num_tokens / max_seq_len); encode() and the HTTP embedding batcher now read one definition.
• LLM.get_kv_cache_load() -> dict, LLM.get_openengine_discovery() -> dict — {} without a generation executor (keeps OpenEngine's tolerant behavior exactly).
• CompletionOutput.spec_dec_counters -> Optional[SpecDecCounters] (_SpecDecCounters renamed public: a public property should not expose a private type; the class is used only inside executor/result.py), CompletionOutput.text_diff_offset -> int.
• SamplingParams.set_request_provided_fields(Iterable[str]) (public rename, no private alias: its only callers are the two frontends), SamplingParams.stop_word_ids property.
• MultimodalDataTracker.multimodal_server_config property.
• OpenAIDisaggServer.address -> tuple[str, int] (OpenAIServer.host/port hold the address passed to __call__, so those names would mean something different).
• PyExecutor.set_openengine_discovery(), PyExecutor.set_openengine_node_server() — the node-discovery hook's two writes; the fields stay private for the executor's own reads.
• registered_model_type(processor_cls) in tensorrt_llm/inputs/registry.py.
• tensorrt_llm/llmapi/engine_client.py — the EngineClient protocol (new module, added to the strict-mypy file list in pyproject.toml).

Names chosen here for the API review committee to confirm: engine_in_worker_process, record_fatal_error, multi_frontend_ipc_dir, hf_model_config, text_diff_offset, set_request_provided_fields, address, set_openengine_discovery / set_openengine_node_server, SpecDecCounters, registered_model_type.

Behavior

No behavior change on any path a working configuration reaches. Four differences exist only on unreachable paths:

1. OpenAIServer._init_llm used a tokenizer-name_or_path fallback when generator._hf_model_dir was falsy; LLM.hf_model_dir falls back to args.model instead. _hf_model_dir is assigned during construction from the model loader and is a directory in every path, so the old fallback never ran.
2. LLM.resource_governor_queue raises RuntimeError without a generation executor where the old inline read raised AttributeError (an encode-only LLM served in the LLM role already failed at that point).
3. OpenEngine start() with KV events configured on an encode-only LLM now raises RuntimeError from LLM.get_kv_cache_capacity() where it raised AttributeError before; that configuration already failed at startup.
4. GetServerInfo's capacity query goes through LLM.get_kv_cache_capacity(); after llm.shutdown() that raises where the old tolerant getattr chain silently skipped the block. Unreachable: OpenEngineServer.stop() runs before llm.shutdown() in the launcher, so no RPC is served once the executor is gone. (Encode-only LLMs are handled by the explicit args.encode_only guard, exactly as the old executor-less path skipped the block.)

LLM.resource_governor_queue reads self._executor directly rather than through getattr(..., None): the api-stability snapshotter (ClassSnapshot.from_inspect) evaluates every attribute on an attribute-less __new__ instance and tolerates only AttributeError, so the property raises AttributeError on a bare instance and RuntimeError only on a built LLM whose executor is gone; a test pins the bare-instance behavior.

Moves

None.

CI bookkeeping

• Multi-GPU table: none (no new directories; changed files are already covered).
• CODEOWNERS: none.
• legacy-files.txt: none (no new Group B files; tensorrt_llm/llmapi/engine_client.py and the new test file are Group A).
• Test lists: tests/integration/test_lists/test-db/l0_cpu.yml gains unittest/serve/test_serve_runtime_api_calls.py (next to the other serve entries).
• pyproject.toml [tool.mypy] files gains tensorrt_llm/llmapi/engine_client.py (triggers the OSS compliance scan).

api-stability

tests/unittest/api_stability/references/llm.yaml gains four methods (record_fatal_error, multi_frontend_attach_info, get_kv_cache_load, get_openengine_discovery) and six properties (encode_limits, engine_in_worker_process, hf_model_config, hf_model_dir, multi_frontend_ipc_dir, resource_governor_queue); completion_output.yaml gains spec_dec_counters and text_diff_offset; sampling_params.yaml gains the set_request_provided_fields method and the stop_word_ids property. api_stability_core.py adds Iterable to its collections.abc import so the reference annotation evaluates. references_committed/ is unchanged, so only the non-committed committee reviews; please apply the api-compatible label.

In-flight PRs on the same files

• [None][feat] Support image inputs in OpenEngine gRPC #20446 / [None][feat] Support video inputs in OpenEngine gRPC #20654 / [None][feat] Support audio inputs in OpenEngine gRPC #20655 (image/video/audio in OpenEngine): merged; their sites were re-located and the two private reads [None][feat] Support audio inputs in OpenEngine gRPC #20655 added in grpc/openengine/capabilities.py are converted here.
• [https://nvbugs/5577178][fix] Score every prompt token when prompt logprobs hit the KV cache #20646 (prompt logprobs with KV-cache hits): open; touches grpc/openengine/formatting.py — textual-conflict risk only, whichever lands second rebases (_token_holdback now reads the public stop_word_ids).
• [#18151][feat] Admit trusted KV hints and carry immutable request metadata #20269 (trusted KV hints): open; touches llmapi/llm.py and references/llm.yaml — additive on both sides, whichever lands second rebases.

Test Coverage

New and extended tests (all CPU, registered):

• tests/unittest/serve/test_serve_runtime_api_calls.py (new, cpu_only, registered in l0_cpu.yml): production OpenAIServer methods on stand-in objects — the embedding batcher takes encode_limits's three limits; register_routes builds the resource governor from the public queue and skips it for None; _init_llm feeds hf_model_dir to AutoProcessor.from_pretrained and load_pretrained_config; _record_generator_termination classifies worker-process vs in-process engines and stays silent for doubles without the member.
• tests/unittest/llmapi/test_llm_public_runtime_api.py (extended): all new LLM members on object.__new__(LLM) objects with real executor classes — attach-info payload/three ValueError cases with exact text, IPC dir (including the attached-frontend proxy without the field), engine_in_worker_process for both proxies and in-process executors, hf_model_dir fallback, hf_model_config, governor-queue forwarding and RuntimeError, encode_limits and encode() rejecting an oversized batch, load/discovery forwarding and empties, record_fatal_error first-one-wins and no-executor no-op, and the EngineClient conformance tests (member kinds and generate_async parameter set/defaults).
• tests/unittest/llmapi/test_executor.py: CompletionOutput.spec_dec_counters (None default, set value, result-path identity) and text_diff_offset (= len(text) - len(text_diff)).
• tests/unittest/llmapi/test_sampling_params.py: set_request_provided_fields keeps only sampler fields and gates model defaulting exactly as before; stop_word_ids is None before _setup and the encoded ids after.
• tests/unittest/llmapi/apps/test_chat_utils.py: multimodal_server_config returns the configured object and a default when none was given.
• tests/unittest/disaggregated/test_openai_disagg_service.py: OpenAIDisaggServer.address is the configured (hostname, port).
• tests/unittest/_torch/pyexecutor/test_py_executor.py: the two OpenEngine setters store what the getter and shutdown read.
• Frontend doubles updated to the public members in: test_reasoning_prompt_resolution.py, test_openai_disagg_service.py, test_openai_server_iteration_stats.py, test_anthropic_routes.py, openai_server_stub.py, test_serve_multi_frontend_helpers.py (with a new already-shut-down case), test_control.py, test_node.py, test_openengine.py, test_multimodal.py, test_frontend_lifecycle.py.

Commands and results (CPU-only dev container, no GPU):

python tests/unittest/serve/test_private_access_lint.py --report | awk -F'\t' '$6=="cross"' | wc -l   -> 19 (0 unresolved)
pytest tests/unittest/serve/test_private_access_lint.py tests/unittest/serve/api/visual_gen/test_private_access_lint_visual_gen.py tests/unittest/grpc/openengine/test_private_access_lint_openengine.py -q   -> 14 passed
pytest tests/unittest/llmapi/test_llm_public_runtime_api.py tests/unittest/serve/test_serve_runtime_api_calls.py tests/unittest/serve/test_serve_health.py -q (with tests/unittest/api_stability)   -> 122 passed
pytest tests/unittest/llmapi/test_executor.py tests/unittest/llmapi/test_sampling_params.py tests/unittest/llmapi/apps/test_chat_utils.py tests/unittest/llmapi/test_serve_multi_frontend_helpers.py -q   -> 177 passed; 3 failures need LLM_MODELS_ROOT model weights and fail identically on the base branch (test setup, not this change)
pytest -m cpu_only tests/unittest/grpc/openengine tests/unittest/_torch/pyexecutor/test_py_executor.py -q   -> 698 passed, 0 failed (clean serial run)
pytest tests/unittest/api_stability -q   -> 64 passed; references_committed/ unchanged
pytest -m cpu_only tests/unittest/disaggregated -q (clean run: etcd available, LLM_MODELS_ROOT set)   -> 1412 passed, 0 failed (base branch: 1411 passed, 0 failed; the extra test is the new address test)
LLM_MODELS_ROOT=... pytest -m cpu_only tests/unittest/llmapi/apps -q   -> 2261 passed, 1 failed (test_render_standalone.py CUDA-stub startup test, needs the trtllm-render entry point; the same single failure on the base branch)
mypy (pyproject [tool.mypy] files, now incl. tensorrt_llm/llmapi/engine_client.py)   -> Success: no issues found in 17 source files
pytest -rs --collect-only tests/unittest/llmapi/apps/test_harmony_parsing.py tests/unittest/llmapi/apps/test_harmony_channel_validation.py   -> 79 collected, no import skips
fastapi 0.136.3 and 0.139.2: register_routes/serve-health/anthropic-route tests   -> 63 passed on each

Property-based check (Hypothesis) on the request-field recording path: recording equals the sampler-field intersection for arbitrary name lists, and model defaulting fills exactly the unrecorded, non-null config fields; four planted bugs (recording junk fields, inverting the recorded check, letting a JSON null override, never applying defaults) were each detected by the properties.

GPU coverage is left to CI: the embeddings e2e (test_encode_batcher.py), the multi-frontend tests, test_e2e.py::test_openai_health, the disagg e2e blocks in l0_a10.yml, and unittest/grpc/openengine/test_capability_conformance.py.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why.
• PR title follows the convention: [JIRA ticket/NVBugs ID/GitHub issue/None][type] Summary of the change.
• I have performed a self-review of my own code.
• Test cases are provided for new code paths (see test instructions)
• Any new dependencies have been scanned for license and vulnerabilities
• CODEOWNERS updated if ownership changes
• Documentation updated as needed

🤖 Generated with Claude Code