Properly enable the hermetic C++ toolchain

[j2kun]

Contributors have noted that I failed to fully enable the hermetic LLVM build.

#3045 is a prototype fix, and I am going to try to whittle it down to a minimal set of improvements to allow us to properly enable the hermetic build.

One big obstacle is the python frontend shells out to clang on the CLI, and the hermetic toolchain doesn't make that easy.

[j2kun]

OK now I see the issues with the frontend. Let me try to summarize.

1. We currently build libopenfhe.so standalone so it can be distributed with the frontend PyPI package. This libopenfhe.so is built by bazel, and hence with its hermetic C++ toolchain.
2. This hermetic toolchain differs from what is on the user's system. In particular, it differs from what is on GH actions CI (gcc) in terms of stdlib and ABI.
3. However, the frontend's jit-compiler shells out to the system's clang, which discovers system stdlibs and those differ from the hermetic build.
4. Due to the above, the libopenfhe.so ABI and the jit-compiled code ABI differ.

The right solution is, in my view, to stop shipping openfhe with the PyPI backend. Instead, require the user to install openfhe as per their system's C++ setup. Then it will be compatible with the jit-compiler. This further requires the frontend to configure HEIR to point to libopenfhe.so, or else put it in a place that it is automatically discovered by their C++ compiler.

This would also require the CI to install openfhe as a separate step from building HEIR's frontend and running tests, which I think implies that we can't run the frontend tests in bazel.

Thoughts, @AlexanderViand ?

[AlexanderViand]

Doesn't openfhe have a pip install openfhe? Any chance we can just make that an optional dependency?

[j2kun]

That is a good point. I should try using a pip dependency first.

[j2kun]

One small problem: OpenFHE's PyPI package does not support Mac.

[edit]: updating the wheel in #3058 to actually prevent macos from using openfhe's PyPI wheel, and instead it asks the user to compile OpenFHE from source and provide the relevant paths to libs

[j2kun]

The problems are mounting...

Openfhe publishes two classes of wheels: versions suffixed with 22.4 (built on Ubuntu 22.04) and versions sufficed with 24.4 (built on Ubuntu 24.04). The 22.04 version has a good glibc (2.34) but is only compatible with Python 3.10's ABI, so we can't use it with any newer Python. The 24.04 version has a newer glibc (2.38) and Python 3.12's ABI, but manylinux doesn't have any containers with that new of compatibility for x86 (cf. https://github.com/pypa/manylinux#manylinux_2_39-almalinuxrockylinux-10-based---alpha).

I don't think we can pin ourselves to Python 3.10, so that leaves us with these options:

1. Stop shipping OpenFHE and require the user to install openfhe and provide all the necessary info to configure it so heir-py can use their system's compiler. Set up a CI for HEIR that does the same and skips the bazel build of openfhe in favor of a CMake one, and invoke this during the release as testing.
2. Find a way to force the glibc/ABI of the bazel-hermetic openfhe build to be sufficiently old and/or compatible that newer ones can use it without a problem. (Is this even possible?)
3. Publish our own, properly compatible OpenFHE PyPI packages.
4. Work with the OpenFHE team to try to fix their PyPI packages.
5. Wait until manylinux gets a 2.38 x86 container and limit heir-py compatibility

Honestly, I feel that with all these difficulties, the most sustainable path forward is option (1).

[AlexanderViand]

That's annoying, especially after all the time you invested, but it does look like option 1 is best.

You might still want to report the issues to the OpenFHE team? If they fix their package, we can always add it back to the optional dependencies later.