feat(generators): modular cascade rules, grader scaffold, evals#52
Open
iddocohen wants to merge 2 commits into
Open
feat(generators): modular cascade rules, grader scaffold, evals#52iddocohen wants to merge 2 commits into
iddocohen wants to merge 2 commits into
Conversation
Deploying infrahub-skills with
|
| Latest commit: |
165f96a
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://b1fd73c1.infrahub-skills.pages.dev |
| Branch Preview URL: | https://ic-feat-generator-enhancemen.infrahub-skills.pages.dev |
Add nine new rules to infrahub-managing-generators covering modular-generator best practices from the Infrahub docs, plus a grader scaffold and five eval tasks that verify the rules empirically. Universal tier (always apply): - validation-upstream-counts: raise on count mismatch before any .create() to avoid silent partial-fabric corruption via the tracking system's delete_unused_nodes=True semantics. - python-stable-iteration: wrap create-loops in sorted() (or range()) so re-runs produce identical downstream output. - patterns-common.md batch example updated to match the new stable-iteration rule. Cascade tier (apply only when building a modular cascade): - cascade-one-layer: each generator owns one hierarchy level. - cascade-target-inheritance: downstream nodes inherit GeneratorTarget so they carry the checksum attribute. - cascade-checksum-guard: guard saves with a checksum compare (body short-circuits or contains the gated create/save). - cascade-version-constant: GENERATOR_VERSION prefixed into the hash forces re-cascade on logic changes. SKILL.md: new workflow Step 2 makes the single-vs-cascade topology choice explicit so cascade machinery is opt-in rather than silently applied. Step 8 (checksum guard) only runs on the cascade branch. Troubleshooting reference (peer of examples.md, not under rules/ per the documented three-tier architecture): symptom→first-check table and post-modification verification checklist for diagnosing misbehaving cascades. Grader scaffold at graders/managing-generators/ uses AST-based structural checks — accepted patterns are verified against compliant fixtures; violating fixtures are confirmed to fail with named assertions. PyYAML is a hard dependency. The check for upstream-count-validation requires the guard to abort (raise/return), not just mention len(). The check for cascade-checksum-guard requires the guarded body to short- circuit or contain the gated create/save call, rejecting presence-only mentions. Five new eval tasks: generator-basic, generator-validation, generator-deterministic, cascade-upstream, cascade-downstream. skillgrade --smoke scored 5/5 on every task — 25 trials, 75 grader checks, 100% pass rate. dev/guides/running-evals.md gains a Local Setup note explaining that grader scripts invoke `python` (not `python3`) and require an activated .venv on macOS where Homebrew omits the `python` alias. CI is unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add six rules surfaced by reviewing the opsmill/infrahub + opsmill/infrahub-sdk-python source for production bottlenecks the existing skill was silent on. Each rule cites the SDK file:line that motivates it. Tracking discipline (CRITICAL — silent-corruption gap): - tracking-upstream-update-group-context: when generate() modifies a node fetched via .get() / .filters(), the .save() must pass update_group_context=False. The SDK's TRACKING mode auto-enrolls the node in this generator's CoreGeneratorGroup, and on the next run delete_unused_nodes=True will delete the upstream node the generator only intended to modify. Cited: infrahub_sdk/node/node.py:973-1000. Code discipline under python- prefix (HIGH–MEDIUM): - python-no-broad-except: catching Exception without re-raise lets the tracking context exit "clean" with partial in-memory state, becoming the new ground truth on the next run. - python-no-early-return: same failure mode for `return` after .create()/.save() — partial commit. - python-no-self-read-after-create: don't .get(kind=X) after a .create(kind=X) in the same generate(). N+1 cost plus read-replica staleness; the local SDK object is canonical. - python-filters-parallel: client.filters() defaults to sequential pagination (pagination_size=50). For >50-item fan-outs pass parallel=True. Cost is zero on small result sets. - python-kind-literal: every kind= must be a string literal. Closes a known evasion path on cascade-one-layer (which skips non-literal kinds) and surfaces multi-layer-generator antipatterns. Six new AST-based structural check functions in graders/managing-generators/lib.py with corresponding CHECKS registry entries. Verified against compliant + violating fixtures: each rule's fail case scores < 1.0 with a named assertion failure; the compliant fixture scores 1.0 across all eight checks. One new eval task (generator-upstream-modification) exercises the upstream fetch-and-modify pattern — primarily exposing tracking-upstream-update-group-context and no-self-read-after-create. The other rules are advisory checks that pass trivially on existing task prompts; they're wired into every existing per-task grader script as regression coverage so a future refactor can't accidentally introduce a violation in any of the universal or cascade tasks. SKILL.md additions: rule-categories table bumps Tracking from HIGH to CRITICAL; Python Class description expanded; Step 4 (Implement the Python class) gains pointers to all five new python- rules; Step 7 (Make it idempotent) gains the update-group-context guidance. _sections.md reflects the same. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iddocohen
force-pushed
the
ic-feat-generator-enhancement
branch
from
382a761 to
165f96a
Compare
github-actions
Bot
added
the
skill/managing-generators
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add nine new rules to infrahub-managing-generators covering modular-generator best practices from the Infrahub docs, plus a grader scaffold and five eval tasks that verify the rules empirically.
Universal tier (always apply):
Cascade tier (apply only when building a modular cascade):
SKILL.md: new workflow Step 2 makes the single-vs-cascade topology choice explicit so cascade machinery is opt-in rather than silently applied. Step 8 (checksum guard) only runs on the cascade branch.
Troubleshooting reference (peer of examples.md, not under rules/ per the documented three-tier architecture): symptom→first-check table and post-modification verification checklist for diagnosing misbehaving cascades.
Grader scaffold at graders/managing-generators/ uses AST-based structural checks — accepted patterns are verified against compliant fixtures; violating fixtures are confirmed to fail with named assertions. PyYAML is a hard dependency. The check for upstream-count-validation requires the guard to abort (raise/return), not just mention len(). The check for cascade-checksum-guard requires the guarded body to short- circuit or contain the gated create/save call, rejecting presence-only mentions.
Five new eval tasks: generator-basic, generator-validation, generator-deterministic, cascade-upstream, cascade-downstream. skillgrade --smoke scored 5/5 on every task — 25 trials, 75 grader checks, 100% pass rate.
dev/guides/running-evals.md gains a Local Setup note explaining that grader scripts invoke
python(notpython3) and require an activated .venv on macOS where Homebrew omits thepythonalias. CI is unaffected.