[BUG] LiteMP FP16 mixed precision cannot be deployed via qnn-onnx-converter on QAIRT 2.33.0

[charmway]

Summary

We use AIMET-ONNX LiteMP (flip_layers_to_higher_precision(..., override_precision=float16)) to quantize an activation-sensitive RT-DETRv2 detector. The AIMET simulation is essentially lossless (mean AP50 0.4996 vs FP32 0.4987). However, none of the available encoding-export strategies produce a correct mixed-precision context binary through qnn-onnx-converter on QAIRT 2.33.0.

Hard constraint: our production runtime is QAIRT 2.33.0.250327 and cannot be changed. So solutions that rely on Quantizer V2 / encoding 2.0.0 (which only exist in QAIRT >= 2.37) are not viable for us. We are looking for a way to deploy AIMET FP16 mixed precision on 2.33 specifically, or confirmation that it is impossible.

This is closely related to #4090, but that issue's accepted fix (encoding_version="2.0.0" + --use_quantize_v2) requires a newer QAIRT and therefore does not apply to us.

Environment

• AIMET-ONNX: 2.31.0 (+cu126)
• QAIRT / QNN SDK: 2.33.0.250327 (production-locked, cannot upgrade)
• HTP target: v75 (SoC 90) — project "z03"
• Python 3.10
• Model: RT-DETRv2 PResNet-50vd detector (ONNX opset 16, static shapes, contains GridSample / deformable cross-attention). Inputs: images [1,3,640,640] f32, orig_target_sizes [1,2] int64. Outputs: labels/boxes/scores.

Why FP16 is required (QuantAnalyzer)

check_model_sensitivity_to_quantization shows the model is activation-quantization sensitive, not weight sensitive:

Config	eval
FP32	0.1232
weight-only quant (W8, A=float)	0.1228 (≈ lossless)
activation-only quant (W=float, A16)	0.0038 (collapse)

Per-layer activation encoding ranges identify a handful of coordinate activations in the decoder with astronomically large dynamic range, e.g.:

/model/decoder/Add_output_0                 range ≈ 2.7e34   (anchors add)
/model/decoder/decoder/Div_output_0..Div_5  range ≈ 1e5      (bbox log-scale wh)
/postprocessor/Mul_2_output_0               range ≈ 4.6e3    (box coords × image size)

INT8/INT16 cannot represent these, so plain w8a16/w16a16 collapse to AP50 ≈ 0.02. LiteMP keeping these (and the SQNR-sensitive layers) in FP16 restores the AIMET-sim accuracy to ≈ FP32. The problem is exporting that mixed precision to a 2.33 context binary.

Baseline (full eval set, 1154 images, 3-class AP50)

Config	subject	crop	debris	mean	note
FP32	0.7038	0.5821	0.2103	0.4987	reference
fp16 (AIMET sim)	0.7047	0.5848	0.2112	0.5002	~lossless
w16a16 plain PTQ	0.0151	0.0179	0.0000	0.0110	collapse
w8a16 plain PTQ	0.0327	0.0304	0.0000	0.0210	collapse
LiteMP w8a16 + fp16 flip (AIMET sim)	0.7043	0.5834	0.2112	0.4996	target to deploy

The three export strategies and how each fails on 2.33

LiteMP flips sensitive layers to FP16 by setting q.enabled = False. We then sim.export(..., encoding_version=...) and run:

qnn-onnx-converter \
  --input_network model.onnx \
  --quantization_overrides model.encodings \
  --input_list input_list.txt \
  -o model.cpp

1. Omit-style (default 0.6.1 export — FP16 layers absent from encodings)

Converter succeeds, but the FP16-intended layers are silently quantized to 8-bit (QNN default for tensors without an override). Inspecting the generated cpp:

• images input tensor → QNN_DATATYPE_UFIXED_POINT_8 with real scale=0.00392 (1/255), offset=0 — actually 8-bit quantized, not FP16.
• 34 activations that LiteMP intended as FP16 (incl. /model/decoder/Add_output_0, Concat_3, Mul_output_0, enc_output_proj/Add) come out as UFIXED_POINT_8.
• Only 1 NATIVE FLOAT_32 tensor remains.

Converter log confirms the fallback semantics:

INFO - Skipping quantization, no input_list provided      # when no list
INFO - Processed N quantization encodings                  # encodings read
# but tensors absent from encodings -> default 8-bit, not fp16

So the deployed model is effectively all-INT8 on the sensitive layers → on-target accuracy will be far below the simulated 0.4996. This matches #4071 ("when the encodings file is missing encodings for some operators, QNN falls back to computing/默认 quantization").

2. Explicit dtype:float (force FP16 entries in 0.6.1)

To stop the silent a8 downgrade, we set the flipped quantizers to float dtype so they export as {"bitwidth":16,"dtype":"float"} (like MMP/choose_mixed_precision does), skipping initializers / onnx:: constants to avoid putting overrides on non-quantizable tensors.

Converter fails:

ERROR - Encountered Error: setQInfo: Can't set quantization data with data type:
  QNN_DATATYPE_UFIXED_POINT_8 for: onnx::Add_4934 with quantizable flag 0
...
File ".../backend/qnn_quantizer.py", line 435, in quantize
    quantizer.mixed_precision_processing()
ValueError: setQInfo: Can't set quantization data with data type:
  QNN_DATATYPE_UFIXED_POINT_8 for: onnx::Add_4934 with quantizable flag 0

Key detail: onnx::Add_4934 is an initializer constant (the bias-like addend of conv1_1/norm/Add) and is absent from both the omit and the float encodings. The omit export converts fine; only the float export triggers this error. So the FP16 dtype:float entries on other layers change QNN's mixed_precision_processing propagation such that it tries to quantize this constant (flag 0) → crash. This looks like the same V1 float-fallback / mixed-precision-propagation bug noted in #4090 as fixed only in Quantizer V2.

3. Encoding 2.0.0 (to trigger Quantizer V2, as recommended in #4090)

sim.export(path, prefix, encoding_version="2.0.0",
           export_int32_bias=True, force_activation_as=None)

The 2.0.0 file is a single encodings list (1173 entries, output_dtype/y_scale fields, no param_encodings/activation_encodings keys).

QAIRT 2.33 qnn-onnx-converter cannot parse it:

ERROR - Node /model/backbone/conv1/conv1_1/conv/Conv: 'param_encodings'
File ".../converters/common/converter_ir/op_adapter.py", line 1338, in update_param_quant_overrides
    if graph.has_user_quantization_overrides() and \
       self.name in graph.user_quantization_overrides['param_encodings']:
KeyError: 'param_encodings'

i.e. 2.33's converter hardcodes the old param_encodings schema key and has no Quantizer-V2 / 2.0.0 support at all. (We confirmed by grepping the SDKs: --use_quantize_v2 and IrQuantizerV2 in qnn-onnx-converter exist only in 2.37/2.41/2.43, not 2.33.)

Questions

1. On QAIRT 2.33.0, is there any supported way to deploy AIMET FP16 mixed precision (LiteMP / manual fp16) through qnn-onnx-converter such that the flipped layers actually run in FP16 (not silently re-quantized to 8-bit, and without the setQInfo ... flag 0 crash)?
2. For strategy 1 (omit-style), is there a converter flag (e.g. a float-fallback option) that makes 2.33 keep encoding-less tensors in float instead of defaulting them to 8-bit?
3. For strategy 2, is the setQInfo: Can't set ... for onnx::Add_4934 with quantizable flag 0 error avoidable on 2.33 (e.g. by excluding specific tensor classes from the overrides), or is it inherently the V1 mixed_precision_processing bug?
4. Is there a recommended 2.33-compatible encoding format/flag combination for mixed INT/FP16 at all, or is QAIRT >= 2.37 strictly required for this?

Any guidance for the 2.33-locked case would be greatly appreciated.

[michaelgtuttle]

Hi cwwu (@charmway), thanks for the detailed description of your issue. It is not impossible to deploy a mixed w8a16+FP16 model in QAIRT 2.33, but mixed precision support in QAIRT has improved a lot since 2.33, so if you are locked to that version you will probably need to do more manual iteration and reconfiguration to work around the issues you're seeing. In some cases, it may be easier to adjust your input model to make it more friendly to pure w8a16 quantization rather than try to work around these compilation failures (assuming you have confirmed w8a16 compiles successfully first). Let me start by answering the specific questions you raised, then provide some suggestions on this front.

Q1

On QAIRT 2.33.0, is there any supported way to deploy AIMET FP16 mixed precision (LiteMP / manual fp16) through qnn-onnx-converter such that the flipped layers actually run in FP16 (not silently re-quantized to 8-bit, and without the setQInfo ... flag 0 crash)?

The recommended way to prevent QNN from inserting missing encodings is to pass --float_fallback to qnn-onnx-converter. This tells qnn-onnx-converter to treat missing encodings as float (this may or may not get past the setQInfo error you saw). You can also drop --input_list since it will not need to run calibration.

Q2

For strategy 1 (omit-style), is there a converter flag (e.g. a float-fallback option) that makes 2.33 keep encoding-less tensors in float instead of defaulting them to 8-bit?

Yes, our recommendation is to always use the --float_fallback flag if you intend to run some layers in fp16

Q3

For strategy 2, is the setQInfo: Can't set ... for onnx::Add_4934 with quantizable flag 0 error avoidable on 2.33 (e.g. by excluding specific tensor classes from the overrides), or is it inherently the V1 mixed_precision_processing bug?

This is likely an inherent quantizer V1 bug (I haven't seen this specific error before). It may be fixed by using --float_fallback, but if not, it is probably still avoidable through some manual effort and experimentation.

Since you mentioned this is a conv output norm op, which type of norm is it? If it is a decomposed BatchNorm op, it is worth investigating why it gets decomposed at export (e.g., if it is a custom BN implementation). A standard BatchNormalization op can likely be folded into the adjacent conv which would bypass the problematic tensor altogether. If it is a decomposed LayerNorm, re-exporting to onnx with opset 17+ will result in a single fused operator.

Q4

Is there a recommended 2.33-compatible encoding format/flag combination for mixed INT/FP16 at all, or is QAIRT >= 2.37 strictly required for this?

--float_fallback --float_bitwidth 16 are the key flags, though it still does not guarantee that the model can compile in all cases. In some cases, manually reconfiguring surrounding tensors to float16 can help get past compilation failures. Some other recommended flags that are not related to float mixed precision are --bias_bitwidth 32 (quantizes bias for integer kernels to int32), --use_dynamic_16_bit_weights (allows int16 x int16 matmul), --enable_per_row_quantized_bias (necessary if you have per-channel quantized matmuls)

Model adaptation guidance

If you are not having luck with the above recommendations, it may be easier (and faster on-target) to see if you can adapt your input model to work in w8a16 (or at least reduce number of float16 tensors). For example if you look at the sensitive tensors you mentioned:

• /model/decoder/Add_output_0 (range ≈ 2.7e34): where is this large value coming from? Often when we see a huge range on the output of an Add operation, it is applying some type of masking to the input tensor, and we can usually significantly reduce the value of this mask without degrading accuracy. You can try iterating on the float value to see how small you can make this without degrading the fp32 output.

• /model/decoder/decoder/Div_output_0..Div_5 (range ≈ 1e5): Often Div op ranges can explode if you have some elementwise division where some values in the denominator tensor can be very small. If your model has some pattern like div_output = Div(numerator, Clip(denominator, min=epsilon)) or div_output = Div(numerator, denominator + epsilon)), you can just increase the epsilon value to improve stability (again, best to iterate on the float model to make sure increasing epsilon does not degrade accuracy).

• /postprocessor/Mul_2_output_0 (range ≈ 4.6e3): If this is just a post-processing step on the output, you may be able to easily move it outside the model to avoid quantization noise.

Let me know if you have any additional follow-up questions. If you haven't done so already, I recommend trying to compile the pure w8a16 model to target as well (with --float_fallback), so we can narrow down which issues are specific to the mixed-precision path.

[charmway]

Hi michaelgtuttle , thank you very much for the detailed guidance — it was extremely helpful and we've made significant progress. Sharing our results in case they're useful.

1. --float_fallback resolved the setQInfo ... flag 0 crash (Q1/Q3).
Adding --float_fallback (and dropping --input_list) got us past the onnx::Add_4934 flag 0 error. To answer your question on the norm type: it was a decomposed BatchNorm (Mul + Add) from fold_all_batch_norms_to_weight; the Add's shift is an initializer constant, and LiteMP flipping that Add output to FP16 was what triggered the crash. We worked around it on our side by re-enabling (keeping quantized) any Add whose input is an initializer constant, but your point about avoiding the decomposition at export is well taken.

2. We took your model-adaptation advice and split the post-processing out of the model.
Following your suggestion on /postprocessor/Mul_2 (box-coords × image-size — pure post-processing), we cut the entire RT-DETR post-processing tail out of the ONNX graph. The new model has a single input images and outputs the raw decoder heads (sigmoid scores [1,300,3] + raw boxes [1,300,4]); box-decode / coord-scaling / TopK now run in fp32 on host. This eliminated the Mul_2 FP32/FP16 boundary mismatch entirely, and we verified the split model + host post-processing is loss-less vs the original (AP50 0.4988 vs 0.4987).

3. Pure FP16 deploys correctly and matches the prototype on-target.

• qnn-onnx-converter ... --float_bitwidth 16 (no encodings, no input_list) → on-device AP50 subject 0.749 / crop 0.507 / debris 0.198, aligned with our FP32 prototype. This is our reliable path on 2.33. ✅

4. The W8A16 + FP16 mixed-precision path compiles and looks correct in the .cpp, but still shows large accuracy loss on the HTP target.
With --quantization_overrides <aimet_encodings> --float_fallback --preserve_io datatype --float_bitwidth 16 --bias_bitwidth 32:

• Conversion succeeds; the generated .cpp looks correct (UFIXED_POINT_16 activations, SFIXED_POINT_8 weights, SFIXED_POINT_32 bias, FP16 on the protected sensitive layers, only ~7 ops fall back to float).
• AIMET QuantSim accuracy is loss-less (AP50 ≈ 0.499).
• But on the HTP device, accuracy drops drastically (effectively to ~0).

We compared an on-device dump against AIMET sim on the exact same input image and confirmed the intermediate activations diverge significantly on-target, even though the AIMET-sim result is correct with the same encoding. Since the sim is fine and only the on-target result diverges, this looks like a V1-quantizer sim-vs-HTP execution mismatch rather than an encoding gap (the missing encodings on the MatMul/LayerNorm intermediates are present in both, and sim handles them correctly).

Questions:

1. For the W8A16 on-target accuracy loss, is there a recommended way to narrow down which quantized layers diverge most on HTP vs sim on 2.33, or specific layer types known to diverge that we should force to FP16?
2. You mentioned --use_dynamic_16_bit_weights and --enable_per_row_quantized_bias — given our weights are per-channel int8 and activations int16, would either be relevant here?

For now we're proceeding with pure FP16 as the production path on 2.33 (verified on-target), and treating W8A16 mixed precision as blocked pending the above. Thanks again for the thorough help!

[charmway]

Hello. Any updates?

[michaelgtuttle]

Hi cwwu (@charmway), sorry for the delayed reply. This type of catastrophic accuracy degradation definitely sounds like a bug during conversion. QAIRT/QNN provides an accuracy debugger to help identify which layer(s) are causing the discrepancy, I would start there:
https://docs.qualcomm.com/doc/80-63442-10/topic/general_tools.html#qairt-accuracy-debugger

Generally if you can find the first layer in the model where you see a large deviation between sim and target, that can help pinpoint what went wrong. A deviation this large is unlikely to be caused by subtle differences between integer vs floating-point kernel math, but probably points to something incorrect at a graph level (e.g., an int16 tensor getting down-converted to int8 incorrectly).

The flags --use_dynamic_16_bit_weights and --enable_per_row_quantized_bias can still come into play in your case, but only if you have MatMuls/Linears in your model. Specifically, --use_dynamic_16_bit_weights is confusingly named but will impact any MatMul in your model between two activation tensors and could cause the type of accuracy drop you are observing. If you do not pass this flag in qnn-onnx-converter, it will force the second input of MatMuls to int8 even if your encoding file specifies int16. This may or may not kill your on-target accuracy depending on how sensitive your MatMul second inputs are to int8 quantization.

Other layers you may want to watch out for are Softmax/Sigmoid/Tanh which have strict constraints on their output min/max ranges and may produce incorrect results if the encodings don't adhere to these depending on the qairt version.