[macOS] Fix Metal RHI 3D viewer crashes and enable the viewer on Apple Silicon

[NeverGET]

Description

Adds macOS support for Meshroom's Qt 3D viewer on the Metal RHI backend. On macOS the Qt Quick scene graph runs on Metal, whose pipeline-state creation strictly validates that every vertex-shader input is satisfied by the geometry's vertex descriptor. Several viewer geometries provide only vertexPosition, while the built-in Qt3D materials they use declare vertexNormal — so on Metal the pipeline fails to build and the app crashes. OpenGL tolerates this, which is why it only surfaces on macOS.

This PR makes the viewer build valid Metal pipelines with no behavioral change on other platforms, plus the macOS-specific environment setup needed to run against a PySide6 wheel. It builds on the now-merged macOS support in AliceVision (alicevision/AliceVision#2019). The 3D SfM/camera/depth overlays additionally need a small companion fix in QtAliceVision (plugin rpaths + descriptor normals) — see the linked QtAliceVision PR.

Features list

• SphericalHarmonics material — drop the vertexNormal input; reconstruct a flat normal from dFdx/dFdy of world position in the fragment shader.
• Widget geometries (Grid3D, BoundingBox, Locator3D) — attach a constant normal to satisfy Metal's vertex descriptor, and neutralize the Phong terms (diffuse/specular = black) so the constant normal can't produce wrong shading; widgets render unlit, as before.
• Loaded meshes — Scene3DHelper.ensureNormals() adds default normals to OBJ/PLY geometries lacking them, so built-in lit materials build valid pipelines.
• QML modules — add PySide6's bundled QML path so Qt3D / QtQuick.Scene3D resolve from a PySide6 wheel.
• setupEnvironment — guard the Qt plugin/QML paths behind existence checks; default the RHI backend to Metal on macOS (overridable); rely on the bundle's rpaths for library resolution (no dynamic-linker env var).

Implementation remarks

All changes are additive and platform-guarded — Linux/Windows are unaffected (the shader normal reconstruction is valid on every backend; the env changes are macOS-only).

Addresses the earlier review: the DYLD_FALLBACK_LIBRARY_PATH / AV_BUNDLE handling, the cgroup change, the speculative OpenGL fallback techniques, and the out-of-scope groupDesc commit have all been removed; the branch is rebased onto current develop.

Tested on macOS / Apple Silicon (PySide6 6.10.2, Qt 6.10.2, Metal RHI) against a clean upstream AliceVision develop build: the 3D viewer opens and resizes without crashing, and SfM points + cameras, the textured mesh (EXR), and per-view depth maps all render; the full SfM → DepthMap → Meshing → Texturing pipeline runs.

Thanks to @philippremy for the AliceVision macOS support (#2019) and to the @alicevision team.

[philippremy]

This is AWESOME! I ran into all these issues and did some very dirty hacking to get a bare minimum of it working 😅.
That would have been a follow-up project for me eventually but honestly: I am really not into Qt and QML... you are a lifesaver! I'll make sure to do some testing the following days ^^.
So a big thank you for looking into this!!!

[NeverGET]

You're welcome, BTW I bundled the app and using daily 😄.
I didnt test all the features in the app, i am still investigating the app, if you have a test list for it, i can try on and keep fixing the issues for MacOS

[fabiencastan]

What's the reason to change that?

[NeverGET]

Actually the mesh preview is looking a bit phased off, like whiteish. I tried to make it look like full saturated colour but it didn't worked either and I was feeling very happy about the app is working, I directly go into PR.

Tldr; it just a mess I forgot to revert

[fabiencastan]

Could you fix that?

[codecov]

Codecov Report

❌ Patch coverage is 22.22222% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.33%. Comparing base (a2bb624) to head (0c2672e).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
meshroom/__init__.py	22.22%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3030      +/-   ##
===========================================
- Coverage    85.38%   85.33%   -0.06%     
===========================================
  Files           73       73              
  Lines        11406    11414       +8     
===========================================
+ Hits          9739     9740       +1     
- Misses        1667     1674       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[fabiencastan]

Suggested change

	@deprecated.depreciateParam("group", "Param 'group' on {name} should not be used anymore. Please use 'commandLineGroup' instead")
	@deprecated.depreciateParam("group", "Param argument 'group' on {name} should not be used anymore. Please use 'commandLineGroup' instead")
	@deprecated.depreciateParam("groupDesc", "GroupAttribute argument 'groupDesc' on {name} should not be used anymore. Please use 'items' instead")

[fabiencastan]

@NeverGET Could you rebase this PR onto the develop branch (instead of a merge)?
@philippremy Could you review and test this PR on macos?

[philippremy]

@philippremy Could you review and test this PR on macos?

Will do. I think I have a valid AdaptiveCpp build now and will report any issues asap. I'll be on a train ride tonight, so maybe I can take it for a spin then.

[philippremy]

I just went ahead and did the testing:

(+): It does not crash anymore, models and images are loaded correctly and (afaict) the 3D Viewer now works correctly on macOS (🚀)1

(-): I don't know what to think about the DYLD_FALLBACK_LIBRARY_PATH and AV_BUNDLE parts.

1. Regarding DYLD_FALLBACK_LIBRARY_PATH, using environment variables which directly affect the dynamic linker is really not a good idea and an inherent safety hazard (and I assume it already is/soon will be nonfunctional because of macOS sandbox restrictions and System Integrity Protection). I would rather prefer to modify the relevant libraries (namely AliceVision and QtAliceVision) and set the correct rpaths when building them. For AliceVision, I guess we are already fine since my PR got merged. The resulting bundle folder can just be renamed to aliceVision and works out of the box. Remaining are the libraries for QtAliceVision, which would be a very simple PR I could submit.

2. The AV_BUNDLE environment variable assumes the bundle structure from my heavily-modified MTL-AliceVision fork (i.e., a macOS .bundle folder) afaik. Since the relevant macOS PR got merged over at AliceVision, macOS now also uses a standard Unix root bundle (i.e., lib, bin, share). So I guess this is not needed anymore, since macOS now essentially behaves like Linux.

Everything else LGTM. But I would really prefer to not use/introduce more environment variables but instead try to reduce them in general.

[NeverGET]

Hi @fabiencastan @philippremy — apologies for the long silence on this one. The past couple of months turned out to be unexpectedly busy and I only now got the time to come back and properly address the review. Thanks a lot for the detailed feedback and for taking the time to test it on macOS.

I've pushed an updated branch that addresses every point:

• Rebased onto the latest develop — the merge commit is gone, it's a clean linear history now.
• MaterialSwitcher.qml: reverted the ambient/shininess/specular override. It was a leftover experiment that didn't actually change anything visually, so the textured material is back to using the mesh's own material properties.
• attribute.py: applied the suggested @deprecated.depreciateParam("groupDesc", ...) decorator.
• DYLD_FALLBACK_LIBRARY_PATH: removed. Agreed it shouldn't be set from Python — library resolution should rely on the rpaths baked into the AliceVision bundle, which works now that the macOS PR is merged. Happy to follow up once the QtAliceVision rpath PR lands.
• AV_BUNDLE: removed. Since the macOS support PR was merged, macOS uses the standard Unix bundle layout, so the special-case path discovery is no longer needed.

What remains is the macOS RHI/shader work — Metal normal fixes, OpenGL fallback techniques, the cgroup fix, and the Qt plugin-path existence guards. I re-tested locally on macOS (Apple Silicon, Metal RHI): the full pipeline recomputes end to end and the 3D viewer works.

Let me know if anything else needs adjusting.

[servantftransperfect]

Hello,

Thanks for your contribution.

• I don't think you understood the cgroup usage. This is passed to the command line as a user defined cap on the number of cpu/ max memory to use. The command line are already using the necessary logic to detect the real number of cpus (available memory). This chunk of code is necessary for large shared computers which have per user limitations. It should return -1 if cgroup does not exists. See this link : https://github.com/alicevision/AliceVision/blob/develop/src/aliceVision/system/hardwareContext.cpp .

• Why are you creating alternatives to RHI shaders ? can you describe a real use case ?

Thanks !

[NeverGET]

Hi @servantftransperfect — thanks, both points are fair.

cgroup: you're right, I had the semantics backwards. Looking at the hardwareContext.cpp you linked, --maxCores is meant to express a user-imposed cap, not the detected core count — and the existing OSError-catching path in getCgroupCpuCount() already returns -1 correctly on non-Linux. So the early-return I added was both unnecessary and semantically wrong.

For context on why I touched that file in the first place: while debugging the macOS crashes I also noticed Meshroom wasn't using all CPU threads on my machine, and I tried a few things to nudge it — that change was one of those experiments and ended up being a leftover. With it dropped, AliceVision's own detection takes over, which is what should happen.

OpenGL fallback shaders: no real-world use case. The actual macOS Metal fix is in the modified SphericalHarmonics.vert/frag (RHI path); the parallel _gl technique I added was speculative — it would only kick in if someone explicitly set QT3D_RENDERER=opengl to force Qt3D onto its legacy non-RHI OpenGL backend, which isn't something I had a concrete reason for. Dropped.

Branch is force-pushed; the PR now has 6 focused commits.

[servantftransperfect]

I am not sure to know why, but without your PR, the bounding box and the widgets were correctly lighted using the directional light (in the 3D viewer).

Now using your PR, as you explicitely added wrong normals, the lighting is weird.

I am not sure we wanted (in the first place) those widgets to be affected by the lighting.

Would you have time finding a solution to make them unlit ? I can see in the doc there is a PerVertexColorMaterial which requires to replace normals with color.

[NeverGET]

Hi @servantftransperfect — thanks for the pointer, agreed the uniform (0, 1, 0) normals driving Phong's diffuse/specular terms were producing visually wrong shading. I tried the PerVertexColorMaterial route first since it's exactly what Locator3D already uses, but it turned out worse on macOS / Metal RHI:

• Bare PerVertexColorMaterial (no normals): 4–6 Failed to create render pipeline state: Vertex attribute vertexNormal(1) is missing from the vertex descriptor errors at scene init, then segfault.
• PerVertexColorMaterial + (0,1,0) normals as descriptor placeholder: still 2 such errors, still segfault on viewport resize.

It looks like Qt 6.8's RHI shader generation for PerVertexColorMaterial declares vertexNormal in the vertex stage regardless of whether the fragment shader uses it — Metal validates the descriptor strictly and refuses to build the pipeline. OpenGL (which I believe is your test backend) doesn't enforce this, which is why the suggestion works on your side.

What I landed instead — same intent, different lever:

Keep PhongMaterial on Grid3D and BoundingBox edges (the proven-stable code path) but zero out diffuse and specular:

PhongMaterial {
    ambient: "#FFF"      // grid color, unchanged
    diffuse: "#000"
    specular: "#000"
    shininess: 0
}

With diffuse and specular both black, the Phong equation collapses to just ambient — constant color, normals don't contribute to the shaded result. Visually unlit. The (0, 1, 0) normals stay in the geometry as inert Metal-descriptor plumbing.

Net effect: the lighting weirdness you flagged is gone, and the macOS Metal path stays on shader/material combinations Qt is happy with.

Force-pushed; the PR's second commit is now "Add normals and neutralize lighting on widget geometries" and reflects this.

---

Separate observation (not part of this PR): there's a pre-existing Qt3D-on-Metal crash when the user resizes the 3D viewer. The remaining vertexNormal(1) pipeline failure visible in the macOS log isn't from any QML geometry — by elimination it's coming from the C++ SfMDataEntity / DepthMapEntity plugins (camera frustums / depth-map points), which build their own Qt3D geometries without normal attributes. Resize forces a pipeline rebuild → the broken pipeline state → segfault. This reproduces on the unmodified develop branch as well. Likely best addressed by a small AliceVision-side patch to attach normals to those entities. I can open a separate issue for tracking if helpful.

[servantftransperfect]

Please remove your commit about groupDesc.

• It is not correct (You changed default value where it was not necessary, creating bugs)
• It is not in scope with the other commits.

[NeverGET]

Hi @servantftransperfect, @philippremy, @fabiencastan — thanks for the thorough review and the patience on this one.

I've pushed an update that addresses everything raised and tightens the scope:

Review points

• groupDesc — removed entirely. You were right that it was both incorrect (it changed a default and introduced bugs) and out of scope.
• Widget lighting — kept the PhongMaterial path but with diffuse/specular zeroed, so the constant normals can't drive any visible shading; Grid3D / BoundingBox render unlit again, matching the original look.
• Earlier rounds also dropped the speculative OpenGL fallback technique and the cgroup change you flagged — appreciate you catching those; they had no real use case and didn't belong here.

Housekeeping

• Rebased onto current develop (clean linear history).
• Reworded two commit messages that had drifted from the code — they still mentioned DYLD_FALLBACK_LIBRARY_PATH / AV_BUNDLE, both of which were already removed; library resolution now relies entirely on the bundle's rpaths.

Companion PR
The 3D SfM / camera / depth-map overlays additionally need a small fix on the QtAliceVision side — the plugins link AliceVision via @rpath but installed no rpath of their own, so they failed to load on macOS. I've opened alicevision/QtAliceVision#98 for that: it enables CMAKE_INSTALL_RPATH_USE_LINK_PATH (the build-time rpath approach @philippremy suggested) plus two small descriptor/build fixes. Full credit to @philippremy for both the macOS foundation in #2019 and for pointing me toward the correct rpath fix rather than runtime env vars.

This PR now focuses purely on the Meshroom-side Metal RHI viewer fixes — everything additive and platform-guarded — and I re-tested the full SfM → DepthMap → Meshing → Texturing pipeline plus the 3D viewer on Apple Silicon. Happy to adjust anything further. Thanks again!

[servantftransperfect]

Thank you for this PR !