llm-uservice: Support air gapped environment#1085
Conversation
|
Let's wait for opea-project/GenAIComps#1758, opea-project/GenAIComps#1743 to land in first |
eero-t
left a comment
eero-t
left a comment
There was a problem hiding this comment.
Few typo fixes + text suggestions
bfa4919 to
93b6aea
Compare
|
Not sure if this flag help with the air gapped mode? HF_HUB_OFFLINE=1 |
If I understood correctly, it would help by blocking engines using HF download facilities, from trying to access online resources. Does that mean it having impact only on TEI / TGI, but not on vLLM, Ollama etc? |
HF_HUB_OFFLINE only prevent the huggingface modules to download the model data from online resources. If the data is not existing in local cache dir, it will still cause problems. The only way to use it is when the data is already exiting in local dir, but sometimes the underlying huggingface modules will still try to access online resources which cause the failure. Just like what we did in #1072. However, for llm-uservice, we don't need it. |
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Following GenAIExample to change the default model of vllm as a temporary workaround for GenAIComps issue #1719. Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
for more information, see https://pre-commit.ci
64246ee to
2251d0c
Compare
3 participants
Description
Issues
n/a.Type of change
List the type of change like below. Please delete options that are not relevant.
Dependencies
List the newly introduced 3rd party dependency if exists.
Tests
Describe the tests that you ran to verify your changes.