Skip to content

Reject non-object inputs when validating saved evaluation parameters - #1147

Open
Gareth.Y (linjiaye929) wants to merge 1 commit into
braintrustdata:mainfrom
linjiaye929:fix/reject-nonobject-eval-parameters
Open

Gareth.Y (linjiaye929) wants to merge 1 commit into
braintrustdata:mainfrom
linjiaye929:fix/reject-nonobject-eval-parameters

Conversation

@linjiaye929

Copy link
Copy Markdown
Contributor

RemoteEvalParameters.validate() is a boolean predicate for a parameter object, but its helper calls dict(data) before checking the JSON schema. For a schema with an optional integer limit, this makes [["limit", 2]] validate as if it were {"limit": 2}. An empty array or empty string can also pass, whereas None, True and integers can raise TypeError instead of returning False.

Check for a mapping before normalizing the input; mappings still use the existing defaults and JSON Schema validation. The regression covers all six non-object inputs and valid and invalid object controls. No service, API key or model call is needed.

Verification on 4df5c78d (Linux):

  • All six new cases failed before the change.
  • make fixup: all seven configured hooks passed.
  • make -C py test-core (Python 3.14.7): 1,231 passed, 112 skipped, 12 expected failures; two pytest-asyncio deprecation warnings.
  • The parameter test file also passed with Pydantic available (25 tests on Python 3.12.14).

This fix was prepared with AI assistance. Optional provider integration suites were not run; the change does not alter a provider integration.

@linjiaye929
Gareth.Y (linjiaye929) marked this pull request as ready for review November 19, 2026 04:20
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T04:22:34.086956Z e027eaa Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant