feat(feedback): enrich ChromaFeedbackStore with torchembed metadata embeddings#52
Open
py-ai-dev wants to merge 1 commit into
Open
feat(feedback): enrich ChromaFeedbackStore with torchembed metadata embeddings#52py-ai-dev wants to merge 1 commit into
py-ai-dev wants to merge 1 commit into
Conversation
…mbeddings ChromaFeedbackStore now accepts enrich_with_metadata=True (opt-in, off by default). When torchembed is installed, numerical and temporal fields from each FeedbackEntry — original_score, correct_score, created_at hour and weekday — are encoded into a dense vector and appended to the stored document string. This gives the ChromaDB retrieval index a weak joint signal over both semantic content and score / time proximity, improving few-shot example retrieval for calibration workflows. New components: - fi/evals/feedback/torchembed_encoder.py — FeedbackMetadataEncoder using GaussianFourierProjection (scores) and CyclicEmbedding (hour/weekday) - tests/evals/test_torchembed_encoder.py — 8 unit tests (all passing) Opt-in via new pyproject.toml extra: pip install ai-evaluation[feedback-torch] No changes to the default ChromaFeedbackStore behaviour; existing users are unaffected. torchembed is a zero-dependency PyTorch embedding library (https://github.com/liodon-ai/torchembed) integrated into DeepSpeed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ChromaFeedbackStoreusing torchembed, a PyTorch embedding library integrated into DeepSpeedenrich_with_metadata=True, numerical and temporal fields fromFeedbackEntry(scores + timestamps) are encoded into a dense vector and appended to the stored document, giving retrieval a joint signal over semantic content and score proximity / recencyFalse, new[feedback-torch]optional extraMotivation
ChromaFeedbackStorecurrently embeds only the text ofFeedbackEntry.to_embedding_text(). For calibration workflows this means two entries with identical text but opposite scores (one human-corrected to 0.1, another to 0.9) are indistinguishable at retrieval time. Adding score and temporal context via torchembed'sGaussianFourierProjection(scores) andCyclicEmbedding(hour/weekday) closes this gap.New API
New extra
pip install ai-evaluation[feedback-torch] # installs: chromadb + torchembed + torchFiles changed
python/fi/evals/feedback/torchembed_encoder.pyFeedbackMetadataEncoder+make_enriched_documentpython/fi/evals/feedback/store.pyenrich_with_metadata/metadata_weightparams toChromaFeedbackStorepython/pyproject.tomlfeedback-torchoptional extra; addtorchembedtoallpython/tests/evals/test_torchembed_encoder.pyTest plan
pytest python/tests/evals/test_torchembed_encoder.py— 8/8 passChromaFeedbackStore(enrich_with_metadata=False)(default) is unaffectedChromaFeedbackStore(enrich_with_metadata=True)with a live ChromaDB instance🤖 Generated with Claude Code