feat(trainer): suggest valid HuggingFace model IDs on validation failure#44
feat(trainer): suggest valid HuggingFace model IDs on validation failure#44taeyang0505 wants to merge 2 commits into
Conversation
Signed-off-by: TaeYang Hong <hongtaeyang1231@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
🎉 Welcome to the Kubeflow MCP Server! 🎉 Thanks for opening your first PR! We're happy to have you as part of our community 🚀 Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification! |
There was a problem hiding this comment.
Pull request overview
Adds best-effort HuggingFace Hub-based suggestions to help callers recover from invalid model identifiers during trainer planning flows (notably estimate_resources() and pre_flight()), reducing trial-and-error retries when model ID validation fails.
Changes:
- Added
_suggest_hf_model_ids()to normalize common malformed inputs and query the Hub for close matches. - Attached
suggestionsto the validation error produced by_get_model_info_from_hf(), and surfaced those suggestions throughToolError.detailsinestimate_resources(). - Added unit tests for normalization, lookup-failure fallback, and
_get_model_info_from_hf()error payload shape (Hub mocked).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| kubeflow_mcp/trainer/api/planning.py | Adds Hub-backed suggestion helper and includes suggestions in error details returned by estimate_resources(). |
| kubeflow_mcp/trainer/api/planning_test.py | Adds tests for normalization and suggestion behavior on format-validation failures. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| details: dict[str, Any] = { | ||
| "hint": "Ensure model path is correct (e.g., 'meta-llama/Llama-3.2-1B')" | ||
| } | ||
| if hf_info and hf_info.get("suggestions"): | ||
| details["suggestions"] = hf_info["suggestions"] |
| def test_invalid_model_id_includes_suggestions(): | ||
| """An invalid model ID returns the format error plus suggestions when available.""" | ||
| with patch("huggingface_hub.list_models") as mock_list: | ||
| mock_list.return_value = _fake_models("Qwen/Qwen3-8B") | ||
| result = _get_model_info_from_hf("qwen3:8b") | ||
|
|
||
| assert result is not None | ||
| assert "Invalid HuggingFace model ID format" in result["error"] | ||
| assert result["suggestions"] == ["Qwen/Qwen3-8B"] | ||
|
|
||
|
|
||
| def test_invalid_model_id_omits_suggestions_when_none_found(): | ||
| """When the Hub returns nothing, no empty 'suggestions' key is added.""" | ||
| with patch("huggingface_hub.list_models") as mock_list: | ||
| mock_list.return_value = [] | ||
| result = _get_model_info_from_hf("qwen3:8b") | ||
|
|
||
| assert result is not None | ||
| assert "suggestions" not in result |
…ub lookup Signed-off-by: TaeYang Hong <hongtaeyang1231@gmail.com>
|
Thanks for the thorough review! Addressed both points: _get_model_info_from_hf now also attaches best-effort suggestions when a well-formed but nonexistent ID fails the Hub lookup (e.g. meta-lama/Llama-3), not just on format-validation failure — this covers the second example from the issue. |
What this does
When
pre_flight()/estimate_resources()is called with a malformedHuggingFace model ID (e.g. Ollama-style
qwen3:8b, or a typo likemeta-lama/Llama-3), the response now includes asuggestionslist of validmodel IDs instead of a bare format error, so users and agents can self-correct
without wasted round-trips.
Changes
_suggest_hf_model_ids()inplanning.py: normalises the raw input(
hf://prefix, Ollama:tag,/separators) and querieshuggingface_hub.list_models(search=..., limit=3). Best-effort — returns[]on any lookup error so it never masks the original error.suggestionsto the validation-failure error in_get_model_info_from_hf().suggestionsthrough the publicToolError.detailsinestimate_resources()so they reachpre_flight()callers.planning_test.py) covering normalisation, lookup-failurefallback, and the error payload shape (HF Hub mocked).
Testing
make verifyandmake test-pythonpass locally (150 passed).Closes #40