Skip to content

skill_structure validator false positive — rejects all evolved candidates #110

Description

@solidstatecc

skill_structure validator false positive — rejects all evolved candidates

Problem

The skill_structure constraint validator checks if the evolved skill body contains YAML frontmatter. However, by design, skill["body"] has the frontmatter already stripped. This causes the validator to reject every evolved candidate, including an unmodified baseline, blocking the holdout evaluation step.

Root Cause

load_skill() separates frontmatter from body. The validator then checks the body (with frontmatter stripped) for frontmatter markers — which will always fail.

Evidence

Both Run 01 & Run 02 report:

✗ skill_structure: Skill missing: YAML frontmatter (---), name field, description field
✗ Evolved skill FAILED constraints — not deploying

This blocks holdout eval even for valid, unmodified baselines.

Solution

Check the reassembled skill (with frontmatter re-attached) in the validator instead of the bare body.

Suggested Action

  • Pass reassembled skill to validator's skill_structure check, or
  • Document that skill_structure only applies to reassembled artifacts, or
  • Remove the frontmatter check and verify only during reassembly

Hit this while preparing #109.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions