Skip to content

fix(582): preflight warns when kind:tool UDF may not always emit a required schema field#799

Merged
Muizzkolapo merged 4 commits into
mainfrom
fix/582-udf-conditional-required-field
Jul 17, 2026
Merged

fix(582): preflight warns when kind:tool UDF may not always emit a required schema field#799
Muizzkolapo merged 4 commits into
mainfrom
fix/582-udf-conditional-required-field

Conversation

@Muizzkolapo

Copy link
Copy Markdown
Owner

Summary

Post-568 (flat schema fields required-by-default), a kind: tool action's UDF can declare a field required in the compiled output schema but only emit it inside a conditional branch. The runtime crashes mid-pipeline (_validate_udf_output'X' is a required property) whenever the guard misses. The qanalabs reconstruct_options action hit this at 39/52 records: source_quote was required, but apply_edited_distractors only writes result['source_quote'] = flat['source_quote'] inside if 'source_quote' in flat.

This PR adds a static-AST preflight warning that names action + field before the run starts. Warning-only; agac inspect exit stays 0, consistent with the 566/567/569 preflight family.

Design

agent_actions/validation/udf_required_field_validator.py:

  • Bounded intra-function AST scan — never imports or executes the UDF; source only, via inspect.getsource(function).
  • Recognises unconditional emit shapes: return {"k": ...}, initial result = {"k": ...} dict-literal assignment, top-level result["k"] = ..., top-level result.update({"k": ...}).
  • Errs toward silence — any shape it does not fully recognise (helper-call init, **spread, list return, mid-function reassignment) yields None and no warning. This preserves "0 false positives on tools that unconditionally emit their required fields" (spec's Verification bullet).
  • Tail-return search stops at nested FunctionDef/AsyncFunctionDef/Lambda boundaries so a helper defined below the outer's return cannot shadow it.

agent_actions/services/preflight_service.py adds _collect_tool_required_field_inputs + _warn_tool_conditional_required_field_risks, wired as step 8 after the sibling passthrough check. Reads json_output_schema.required (the tool path always compiles via the openai dialect → dict shape).

Deviations from the spec: the spec listed PreflightService.validate() and/or ValidateUDFsCommand as wire points. ValidateUDFsCommand operates on raw config that has not yet been through schema compilation (no json_output_schema.required to compare against), so wiring landed only in PreflightService. agac inspect and agac run both go through PreflightService, matching the spec's Success metric.

Verification story

  • Harness gates: gate.sh red recorded a genuine assertion failure at the PreflightService.validate() wire point (no ImportError-based fake-RED). gate.sh green re-runs the RED test hash-checked against tampering, then full pytest (8136 passed), ruff format --check, ruff check, mypy agent_actions.
  • HIGH-tier review: 3/3 YES from correctness / blast-radius / tests-anti-gaming blind reviewers. Correctness reviewer flagged a nested-function edge case (a helper defined below the outer's tail return would be picked as the tail via ast.walk); fixed in the follow-up commit with a dedicated regression test (test_nested_helper_return_does_not_shadow_outer_tail).
  • Smoke: skipped — risk.sh reports smoke_required=no; the diff is preflight-only (new validator + wiring), touches no runtime/config-load surface or smoke-registry paths.
  • Real-project verification: prototype detector run against every @udf_tool function in the qanalabs tools/qanalabs-quiz-gen/ tree during development. On apply_edited_distractors the unconditional set is exactly {options, answer, answer_text} (matching the initial dict literal); every other required field is correctly flagged. On tools that emit unconditionally via a full dict literal (e.g. assemble_question), the emitted set covers every required field and no warning fires.

Test plan

  • Assertion-level wire-point RED (tests/validation/test_udf_required_field_wiring.py) proves the warning behavior at the public PreflightService.validate() API without importing the new symbol.
  • Unit suite (tests/validation/test_udf_required_field_validator.py) — 17 cases covering: qanalabs reproducer, unconditional dict literal / via-name / top-level subscript / .update({...}), conditional-only, helper init, **spread, early guard return, list return, subscript-inside-if, additionalProperties: true does not suppress the check, no-required-fields no-op, missing source ignored, syntactically-invalid source ignored, multi-action correctness, nested-helper-does-not-shadow-outer-tail.
  • Full pytest green (8136 passed). mypy agent_actions clean. ruff clean.

🤖 Generated with Claude Code

Muizzkolapo and others added 4 commits July 17, 2026 20:07
Wire-point assertion at PreflightService.validate() proves the missing
warning for a tool whose UDF only conditionally produces a required
output-schema field. Reproduces the qanalabs reconstruct_options mid-run
crash class (post-568 required-by-default).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ma field

Post-568 (flat schema fields required-by-default), a `kind: tool` action
whose UDF only conditionally produces a declared-required field crashes
mid-run with `Output schema validation failed … 'X' is a required property`.
qanalabs `reconstruct_options` hit this at 39/52 records: `source_quote` was
required, but `apply_edited_distractors` emits it only `if 'source_quote' in
flat`.

Add `find_conditional_required_field_risks`: a bounded intra-function AST
scan that collects the set of constant string keys a UDF unconditionally
writes to its returned mapping (initial dict-literal + top-level subscript
assigns + top-level `update({...})`) and warns on every declared-required
schema field not in that set. The check never imports or executes the UDF.
Any construction shape it does not fully recognise (helper-initialised name,
`**spread`, list return, opaque top-level assignment) yields `None` and no
warning — errs toward silence so unconditionally-emitting tools stay clean.

Wired into `PreflightService.validate()` (step 8) alongside the sibling
passthrough check. Warning only; exit stays 0 — consistent with 566/567/569.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_last_return used ast.walk, which descends into nested FunctionDef/Lambda
bodies. A helper defined AFTER the outer's tail return (dead code, but
possible) would provide a return at a higher lineno and shadow the outer's,
silently exonerating conditionally-emitted required fields.

Iterate func.body statements directly, stopping at nested function/lambda
boundaries. Add a regression test.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Static AST checks that reason about a function's OWN control flow must
never use ast.walk on the function node — it descends into nested
FunctionDef/AsyncFunctionDef/Lambda bodies and their returns will shadow
the outer's. Bounded walker starting from func.body is the fix.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@Muizzkolapo
Muizzkolapo merged commit 6bab418 into main Jul 17, 2026
5 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 17, 2026
@Muizzkolapo
Muizzkolapo deleted the fix/582-udf-conditional-required-field branch July 17, 2026 19:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant