Skip to content

refactor(embedding_compute): extract providers into adapter pattern #372

Description

@luojiyin1987

Summary

Split the 1444-line embedding_compute.py monolith into a registry-backed adapter pattern.

Changes

  • embedding_compute.py: 1444 → 403 lines — keeps shared utilities + compute_embeddings() dispatch + register_provider() API
  • New providers/ package — one module per backend:
    • sentence_transformers.py (487 lines)
    • ollama.py (335 lines)
    • openai.py (136 lines)
    • gemini.py (89 lines)
    • mlx.py (82 lines)

Design

compute_embeddings(texts, model_name, mode="sentence-transformers")
    │
    ├─ _init_providers()  → lazy-import all providers
    ├─ _providers[mode]   → dispatch via registry dict
    └─ register_provider("new-mode", fn)  → third-party extensibility

Benefits

  • Adding a new provider: 1 new file + 1 registry line (was: touch 1444-line elif chain)
  • Adding cross-encoder reranker: just register_provider("reranker", fn)
  • Full backward compat — all ~90 callers unchanged
  • Each provider independently testable

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions