Bug: CLI JPEG decoding (stb_image) differs from Pillow and consumes image-family parity margins

[LiRunGuo]

Affected area

Validation or benchmarking

TensorRT-Model-Connect version

main at 393ab02f

Installation method

Development container

Model and revision

Decoder comparison is model-independent. Model impact measured with facebook/dinov2-small@ed25f3a3, facebook/dinov2-base@f9e44c81, facebook/dinov2-with-registers-small@0d9846e5 (Transformers 5.2.0, fp32).

Environment

Ubuntu 24.04 container built from Dockerfile.dev.x86; the stb_image under test is third_party/stb/stb_image.h as compiled into apps/cli/io.cpp; Pillow from the same image.

Minimal reproducer

1. Decode families/dinov3/tests/data/test_img.jpeg with stbi_load(path, &w, &h, &c, 3) (the call in apps/cli/io.cpp:193) and write the raw RGB bytes.
2. Decode the same file with numpy.asarray(PIL.Image.open(path).convert("RGB")).
3. Compare the two arrays; optionally run both through the same image processor and model.

Observed behavior

The native CLI decodes image inputs with stb_image (apps/cli/io.cpp, read_image → stbi_load). The official Transformers/timm references in family E2E tests decode the same file with Pillow (libjpeg-turbo). For JPEG inputs the two decoders return different pixels. So every image-family E2E that feeds a .jpeg to trtmc and to its reference compares two slightly different images, and the family's tolerance budget silently absorbs a difference the family does not own.

The shared maintainer photograph used by many families (families/*/tests/data/test_img.jpeg, 640x382), stb_image stbi_load(..., 3) vs PIL.Image.open(...).convert("RGB"):

• 4.7% of the 733,440 RGB values differ; max absolute difference 3/255; mean 0.059/255.
• A lossless PNG of the Pillow-decoded pixels decodes bit-identically in both.

How much this moves a model depends on the model. Feeding the fp32 Transformers DINOv2 encoders (not TensorRT) the stb-decoded vs Pillow-decoded pixels, through the same BitImageProcessor, changes last_hidden_state by:

checkpoint	relative Frobenius
facebook/dinov2-small	1.60%
facebook/dinov2-base	2.81%
facebook/dinov2-with-registers-small	1.56%

That exceeds the 1% relative-Frobenius gate that the dinov3 family's thresholds use, before any TensorRT error is added. #1406 therefore uses a lossless PNG fixture and asks maintainers to review that choice.

• A family whose native preprocessing exactly reproduces its reference processor (DINOv2's matches BitImageProcessor bit for bit on identical bytes) still cannot demonstrate that through trtmc with a JPEG input.
• Families that currently pass on JPEG may have part of their margin consumed by the decoder, and margins will differ between models for reasons unrelated to them.
• Real users feeding JPEGs to trtmc get features that differ from transformers on the same file by this amount.

Expected behavior

A native trtmc run and the family's official reference should see identical input pixels, or the difference should be a documented part of the validation contract.

Options:

1. Decode JPEG in the CLI with libjpeg-turbo (what Pillow uses), keeping stb_image for other formats. This is a dependency change in the shared application.
2. Keep stb_image and use lossless fixtures (PNG) for image-family parity tests, documenting that JPEG decoding is outside the family contract.
3. Have references decode through the same decoder as the native application.

I have not surveyed which existing families' margins depend on this. I can do that if it is useful.

Relevant logs

image 382x640  decode diff max 3  mean 0.0587  fraction of differing values 0.0468
lossless PNG of the Pillow pixels: stb == Pillow  True

Submission checks

• I searched open and closed issues and found no duplicate.
• This is not a security vulnerability.
• I removed secrets, credentials, private URLs, and restricted artifacts.