Skip to content

feat: add litellm provider#431

Open
RheagalFire wants to merge 2 commits into
NevaMind-AI:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add litellm provider#431
RheagalFire wants to merge 2 commits into
NevaMind-AI:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

📝 Pull Request Summary

Add LiteLLM as a first-class AI gateway provider with both native SDK and HTTP proxy support, enabling access to 100+ LLM providers via pip install memu-py[litellm].


✅ What does this PR do?

  • Adds a new LiteLLMSDKClient that uses litellm.acompletion() and litellm.aembedding() directly (no proxy server needed)
  • Adds a LiteLLMBackend for users who prefer the HTTP proxy path (client_backend="httpx")
  • Registers litellm as a new client_backend option alongside sdk, httpx, and lazyllm_backend
  • Adds litellm>=1.80.0,<1.87.0 as an optional dependency
  • Adds provider defaults: base_url=http://localhost:4000, api_key=LITELLM_API_KEY, auto-selects litellm client backend
  • 11 unit tests covering backend, settings, and SDK client behavior

New files:

  • src/memu/llm/litellm_sdk.py - SDK client with drop_params=True for cross-provider compatibility
  • src/memu/llm/backends/litellm.py - HTTP proxy backend (inherits OpenAILLMBackend)
  • tests/llm/test_litellm_provider.py - 11 unit tests

Modified files:

  • src/memu/app/service.py - registered litellm client backend
  • src/memu/app/settings.py - provider defaults
  • src/memu/llm/backends/__init__.py - registered export
  • src/memu/llm/http_client.py - registered in LLM_BACKENDS and embedding backends
  • pyproject.toml - optional dependency

🤔 Why is this change needed?

LiteLLM provides a unified Python SDK for 100+ LLM providers (Anthropic, Azure, Bedrock, Vertex, Groq, Ollama, etc.). Adding it as a native client backend lets users access any supported provider without running a proxy or waiting for per-provider backends to be added.


🔍 Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactor / cleanup
  • Other (please explain)

✅ PR Quality Checklist

  • PR title follows the conventional format (feat:, fix:, docs:)
  • Changes are limited in scope and easy to review
  • Documentation updated where applicable
  • No breaking changes (or clearly documented)
  • Related issues or discussions linked

📌 Optional

Live E2E (LiteLLM SDK, no proxy, Azure AI Foundry -> Claude Sonnet):

=== LiteLLM SDK E2E (no proxy) ===
Chat model: azure_ai/claude-sonnet-4-6

--- Chat test ---
Response: 4
Model: claude-sonnet-4-6
Tokens: prompt=20, completion=5

--- Summarize test ---
Summary: LiteLLM is an open-source AI gateway that offers a single, unified
interface for accessing and interacting with over 100 different large language
model (LLM) providers.
Tokens: prompt=44, completion=43

Unit tests (11/11 pass):

TestLiteLLMBackend::test_backend_endpoint PASSED
TestLiteLLMBackend::test_backend_name PASSED
TestLiteLLMBackend::test_backend_payload_parsing PASSED
TestLiteLLMBackend::test_backend_summary_payload PASSED
TestLiteLLMBackend::test_backend_vision_payload PASSED
TestLiteLLMSettings::test_defaults PASSED
TestLiteLLMSettings::test_httpx_backend_preserved PASSED
TestLiteLLMSettings::test_preserves_custom_values PASSED
TestLiteLLMSDKClient::test_chat_calls_acompletion PASSED
TestLiteLLMSDKClient::test_chat_omits_api_key_when_none PASSED
TestLiteLLMSDKClient::test_embed_calls_aembedding PASSED

Lint: ruff check + ruff format --check - clean.

Example usage

SDK backend (recommended):

from memu.app import MemoryService

service = MemoryService(
    llm_profiles={
        "default": {
            "provider": "litellm",
            # client_backend auto-selects "litellm" SDK
            "chat_model": "anthropic/claude-sonnet-4-6",
            "embed_model": "openai/text-embedding-3-small",
        },
    },
    database_config={"metadata_store": {"provider": "inmemory"}},
)

HTTP proxy backend:

service = MemoryService(
    llm_profiles={
        "default": {
            "provider": "litellm",
            "client_backend": "httpx",
            "base_url": "http://localhost:4000",
            "api_key": "sk-your-litellm-key",
            "chat_model": "anthropic/claude-sonnet-4-6",
            "embed_model": "openai/text-embedding-3-small",
        },
    },
    database_config={"metadata_store": {"provider": "inmemory"}},
)

@RheagalFire

Copy link
Copy Markdown
Author

cc @sairin1202 @evan-ak

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant