Skip to content

feat: add DeepEval support for single and multi-turn LLM evaluation - #485

Merged
manuelporrasojeda merged 21 commits into
masterfrom
feat/deep_eval
Nov 17, 2026
Merged

manuelporrasojeda merged 21 commits into
masterfrom
feat/deep_eval

Conversation

@manuelporrasojeda

Copy link
Copy Markdown
Contributor

PR Summary

This PR adds optional DeepEval support to Toolium as an additional evaluation framework. Existing SpaCy, Sentence Transformers and OpenAI-based utilities remain available.

Changes

  • Added toolium/utils/ai_utils/deepeval.py with support for:
    • Semantic similarity
    • Answer relevancy
    • Faithfulness
    • Hallucination detection
    • JSON correctness
    • Contextual relevancy
    • Contextual precision
    • Contextual recall
    • Tool correctness
    • Toxicity detection
    • PII leakage detection
    • Summarization
  • Added OpenAI and Azure OpenAI provider configuration using the existing [AI] settings.
  • Added support for JSON objects and arrays from API responses as evaluation context.
  • Added combined contextual evaluation through:
    • evaluate_contextual_with_deepeval()
    • assert_contextual_with_deepeval()
  • Added support for:
    • Evaluating one or multiple contextual metrics in the same call.
    • Shared thresholds.
    • Per-metric thresholds.
    • Metric-specific options through metric_config.
  • Added validation for:
    • Unknown metrics.
    • Missing or invalid thresholds.
    • Required expected_output for contextual precision and recall.
    • Conflicting configuration styles.
  • Added DeepEval to requirements_ai.txt.
  • Added unit tests covering:
    • All supported metrics.
    • OpenAI and Azure configuration.
    • JSON API context.
    • Individual and multi-metric evaluations.
    • Common and metric-specific options.
    • Threshold validation and assertion behavior.
  • Added Azure OpenAI integration tests.
  • Updated existing AI test configuration to use the available Azure deployment gpt-4o-mini.
  • Recalibrated the strict answer-evaluation threshold from 0.95 to 0.90.
  • Updated docs/ai_utils.rst with:
    • Installation and configuration.
    • DeepEval usage examples.
    • Contextual evaluation with one or multiple metrics.
    • metric_config usage.
    • JSON API context examples.
    • Supported options for each metric.
    • Differences compared with the existing OpenAI methods.
    • Threshold and token-usage considerations.
  • Updated CHANGELOG.rst.

Testing

Executed all AI tests excluding the Behave-dependent accuracy tests:

134 passed, 10 skipped in 82.00s

The skipped tests are existing Sentence Transformers tests explicitly marked as skipped because the model is unavailable in the test environment.

Files excluded

uv.lock and the generated .deepeval/ telemetry directory are intentionally excluded from this PR.

Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread requirements_ai.txt Resolved
Comment thread docs/ai_utils.rst Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread toolium/utils/ai_utils/deepeval.py Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread CHANGELOG.rst Resolved
lmcalvo
lmcalvo previously approved these changes Nov 12, 2026

@rgonalo rgonalo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actualiza el nombre de la PR, será lo que se añada automáticamente al changelog

Comment thread CHANGELOG.rst Resolved
@manuelporrasojeda manuelporrasojeda changed the title feat: deepEval tools feat: add DeepEval support for single and multi-turn LLM evaluation Nov 17, 2026
@manuelporrasojeda
manuelporrasojeda requested a review from a team as a code owner November 17, 2026 14:47
Comment thread .gitignore Resolved

@Rcatala18 Rcatala18 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@manuelporrasojeda
manuelporrasojeda merged commit 8e925e7 into master Nov 17, 2026
24 checks passed
@manuelporrasojeda
manuelporrasojeda deleted the feat/deep_eval branch November 17, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants