Repository navigation
Avoid problems with Python dependencies #42
Description
Activity
Is there an (easy) way to detect if the GPU is available and then either pass that as a parameter to the container, or start a different container/config based on that bit. That decision could then be made as part of the code in the apphost.
Is there an (easy) way to detect if the GPU is available and then either pass that as a parameter to the container, or start a different container/config based on that bit. That decision could then be made as part of the code in the apphost.
Not that I know of.
To be honest, after working with the Python-in-Docker solution for a little while, I'm not convinced this leads to a great experience either. The startup time of eShopSupport goes up a lot:
- On the first run, when it builds the Docker image, this is an extremely time-consuming process, taking about 20 minutes on my laptop even with a fast network connection. It's not obvious to the developer what's going on just from the Aspire dashboard. Even though it says the Python dependency is "building", you can still launch the web UIs, and they just behave weirdly (in particular, it doesn't start downloading the Ollama model until the Python image is built, even though these should be completely independenty, and so the chatbot will give "model not found" errors if you try to interact with it).
- On subsequent startups, it takes about 20s for the app to get into a running state. Before we started using Docker for this, it was closer to 5s.
Also when you shut down the app, it leaves the Python container running (presumably because of it being marked as persistent). That's not great either as you then have to tidy up manually.
And for some reason even though the Python dependency is marked as persistent, it never reuses the container across runs for me.
I think the best thing overall would be:
- In the short term, make the Python dependency optional (and default to not being enabled). In this mode, it shouldn't register the Python app resource in Aspire, so none of the above issues would occur. The app would simply not have auto-classification by default, or would do it using an
IChatClient. - Longer term, improve the experience of using small HF models directly on .NET with ONNX or similar, and then run the classifier model directly on .NET (like we already do for embedding models).
Is there an (easy) way to detect if the GPU is available and then either pass that as a parameter to the container, or start a different container/config based on that bit. That decision could then be made as part of the code in the apphost.
Well on windows 11 the nvidia driver installs a command to query it from powershell:
PS > nvidia-smi --version NVIDIA-SMI version : 556.12 NVML version : 556.12 DRIVER version : 556.12 CUDA Version : 12.5 PS >or for more:
nvidia-smior for the works:
nvidia-smi -h
Longer term, improve the experience of using small HF models directly on .NET
Since we're already using a .NET stack here, handle auto-classification (optionally?) via Azure too. In any case, the example is nice, too badk the python dependencies break it more often than not.
Still the getting this issue on mac m3:
Could not fetch URL https://download.pytorch.org/whl/cu118/intel-openmp/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='download.pytorch.org', port=443): Max retries exceeded with url: /whl/cu118/intel-openmp/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) - skipping
ERROR: Could not find a version that satisfies the requirement intel-openmp==2021.4.0 (from versions: none)
ERROR: No matching distribution found for intel-openmp==2021.4.0Maybe there's a way to remove the PythonInference seems that this project doesn't even loaded inside rider:
This would fix #33, #35, and parts of #19
Currently there's a Python dependency, which is only used as an example of being able to host a Python app in Aspire. It's not really essential to the flow of eShopSupport. But this is causing a lot of trouble because (1) you have to restore the
pipdependencies manually; (2) this might or might not succeed, depending on your Python version and OS configuration; (3) even if it does, it will only work at runtime if you have CUDA-compatible hardware.We should make this better. Options: