Skip to content

BUG: Fix defects in V&V'd filters and align their V&V records with the template - #1841

Draft
imikejackson wants to merge 91 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/vv_report_cleanup
Draft

imikejackson wants to merge 91 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/vv_report_cleanup

Conversation

@imikejackson

@imikejackson imikejackson commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR holds all V&V work for the filters that have a V&V record: the V&V policy and templates, the V&V records, and the code fixes for defects found in V&V'd filters. It is a draft. Do not review it yet.

The work stays on one branch until the V&V policy and the report template are stable. This prevents a series of merges that later template changes would make out of date.

V&V policy and templates

  • The V&V reports follow docs/vv_templates/report_template.md exactly.
  • A report is written for a user of develop or of a DREAM3D-NX release. It names DREAM3D-NX release versions, not branches, commits, or PRs.
  • Deviation entries use only the template fields and one optional Evidence paragraph.
  • The sign-off form is <Name>, YYYY-MM-DD; second engineer: <Name>, YYYY-MM-DD. The second engineer is the reviewer who approved the PR with the V&V work. When no approval exists, the sign-off says "second engineer: pending" and the Status is READY FOR REVIEW.
  • The defect-hunt policy and a part for SIMPLNX defects are added to the templates.
  • docs/vv_templates/vv_recertification.md says when V&V work happens after a filter has a V&V record. The trigger rule (does the change alter output for valid input?) is proposed for team review. The single-branch rule and the legacy comparison package rules are agreed.

V&V records

  • All 44 V&V records are rewritten to the template.
  • The DBSCAN record, deviations, and provenance updates move here from BUG: Enforce component shapes and types that filters read #1766.
  • New legacy comparisons (DREAM.3D v6.5.171, a local proof version, and DREAM3D-NX) are recorded for Compute Feature Neighbor C-Axis Misalignments, Compute Feature Neighbors, Compute Feature Centroids, Compute Feature Neighborhoods, and Regularize Z Spacing. The comparison packages are in the V&V working archive.

Fixes in V&V'd filters

Each filter fix is a separate commit with tests that fail on the old code. The fixes are separate from the documentation commits, so they can move to their own PR if necessary.

  • Compute Average C-Axis Orientations: bound Feature Ids at execute time (-76405) and accumulate in double.
  • Compute Feature Reference C-Axis Misalignments: bound Feature Ids (-9806), use the same cell rule for the mean and the standard deviation, and accumulate in double.
  • Compute Feature Centroids from Triangle Geometry: reject empty or mismatched Face Labels (-12903, -12902) and accumulate in double. The exemplar tolerance changes from 0.0001 to 0.05 because the archive holds float32-accumulated centroids; a float64 reference from the archive triangles differs from them by at most 0.0415.
  • Isolate Largest Feature (Identify Sample): fill holes on 2D images.

Status

Test Plan

  • Full test suite passes serially (in-core Release build, PY:: tests excluded).
  • Each changed V&V report matches the report template.
  • Every deviation ID in a report has an entry in its deviations file.
  • Out-of-core test suite passes.

@imikejackson imikejackson self-assigned this Oct 20, 2026
@imikejackson
imikejackson force-pushed the topic/vv_report_cleanup branch 4 times, most recently from a825ad6 to 1d991d7 Compare October 22, 2026 13:28
imikejackson and others added 24 commits October 24, 2026 09:29
An oracle shows that a filter agrees with the fixtures that the engineer
chose. It does not show that those fixtures can find a defect. Issue BlueQuartzSoftware#1758
(AlignGeometries Centroid mode uses the X offset on all axes) shows the
gap: no test ran that mode, and a fixture with equal per-axis offsets would
also pass.

* Add defect_hunt.md: bug-pattern review, mode coverage, symmetry-breaking
  fixtures, mutation check, and metamorphic relations
* vv_policy.md: require the defect hunt after code-path enumeration
* report_gates.md: add the Defect hunt section and row, the symmetry gate
  for oracle fixtures, and the per-mode value-assertion gate
* report_template.md: add the Defect hunt section and row
* oracle_classes.md: add metamorphic relations to Class 4
* Reports that reached READY FOR REVIEW before 2026-10-02 are not
  required to add the section until their next V&V change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add three Part 1 bug patterns: component shape not declared on the
  selection parameter, parent Attribute Matrix not checked, and
  feature-level bounds
* State where each kind of check belongs: structure in the parameter,
  relationships between inputs in preflight, value-dependent bounds in
  execute
* Add a "SIMPLNX vocabulary for findings" section with the Attribute
  Matrix kinds and an example of SIMPLNX wording

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
V&V documents and test comments named DREAM3D 6.5.172, which is a
private proof branch and was never released. Each reference now names
DREAM3D 6.5.171 or describes the locally patched legacy build; no
sentence claims that 6.5.171 contains a fix that only the patched build
had. The FillBadData summary no longer cites the withdrawn three-way
A/B run as evidence. The DBSCAN files are left for BlueQuartzSoftware#1766.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* deviation_template.md: split the file into Part 1 (differences from
  DREAM3D 6.5.171) and Part 2 (SIMPLNX defects and SIMPLNX behavior
  changes). Part 2 entries record the affected released versions and the
  PR and release that fix the defect.
* report_template.md: include defects that an earlier PR fixed in
  "Bugs found and fixed", give the lead sentence for those defects, and
  list only Part 1 entries in the deviations table.
* report_gates.md: add the gates for "Bugs found and fixed".

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* DBSCANFilter.md: keep only the template sections and fields. Remove
  the Phase 2 to Phase 13 working notes. Correct the Algorithm
  Relationship evidence: PR BlueQuartzSoftware#1421 did not change the filter UUID.
* Record Michael Jackson as the second engineer (2026-08-28).
* Give stable IDs to the Random-seed defect (DBSCAN-D7) and the
  all-false-mask defect (DBSCAN-D8), and state the affected releases
  for D5 to D8.
* Record the comparison against DREAM3D 6.5.171 (2026-10-20). It gives
  the same label for every point as the earlier local proof version.
  Put the three-way table in DBSCAN-D1 as its evidence.
* deviations/DBSCANFilter.md: split into Part 1 (D1 to D3) and Part 2
  (D4 to D8), and remove the incorrect note about a UUID change.
* provenance/dbscan_test.md: cite the DREAM3D 6.5.171 run and record
  the second-engineer review.
* The record describes tests and DBSCAN-D6 from PR BlueQuartzSoftware#1766, so merge this
  change after that PR.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* DREAM3D-NX 7.4.2 was released on 2026-08-28, not 2026-08-31.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* Write the legacy application as "DREAM.3D v6.5.171", or "DREAM.3D"
  for short, in all V&V templates. "DREAM3D-NX" stays for the new
  application.
* report_template.md: add the SIMPLNX Human Name header row, list the
  allowed Status values, and add a list of content that does not go in
  a V&V report.
* report_template.md: call a run on a non-stock legacy build "a local
  proof version", and use the DBSCAN report as the worked instance.
* report_gates.md: add the Human Name row and the header-row rules to
  the header gates.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* Add the SIMPLNX Human Name header row.
* Write the legacy application as "DREAM.3D v6.5.171" or "DREAM.3D".

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…late

* Keep only the template sections and fields, and describe the current
  rolling-slice algorithm.
* Map 26 code paths to the 13 current test cases.
* State the release ranges of D1 and D2 from the release branches and
  the DREAM.3D v6.5.171 source.
* Record the 28 of 28 match against DREAM.3D v6.5.171 (2026-08-18).
* Make the provenance sidecar follow the provenance template.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…template

* Keep only the template sections and fields.
* List the 16 current test cases and map 44 code paths to them.
* State the release ranges of D3 to D5 from the release branches and
  the DREAM.3D v6.5.171 source. D3 (2D flat spacing) is still open.
* Remove repeated template blocks from the provenance sidecar.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
… the template

* Keep only the template sections and fields. Fold the sub-sections
  into the sidecars.
* Add stable IDs D3 to D5 for the tolerance, Laue-index and phase-index
  fixes, and state the release ranges from the release branches and the
  DREAM.3D v6.5.171 source.
* Record the DREAM.3D v6.5.171 comparison (2026-09-17): 12 of 28 cases
  match; a local proof version matches all 28.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- Sign-off uses "<Name>, YYYY-MM-DD; second engineer: <Name>, YYYY-MM-DD".
  The second engineer is the reviewer who approved the PR with the V&V work.
- A Legacy comparison must run on a current nxrunner build. Its artifacts
  go to vv_work:<FilterName>/ on OneDrive.
- When no record shows who generated an archive, use the PR author and the
  dates the PR was open, marked as inferred.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- Use one sign-off form in DBSCAN, ErodeDilateMask, ComputeFeatureSizes,
  and BadDataNeighborOrientationCheck.
- ErodeDilateMask: Nathan Young approved PR BlueQuartzSoftware#1717, so the record is
  COMPLETE. Remove stale "this branch" text and an outdated review note.
- Fill in the inferred archive provenance for all three pilot archives.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…he filter class

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
A reader of a V&V report runs develop or a released DREAM3D-NX. Branch
names, commit hashes, and PR numbers do not tell that reader whether
their version has a defect.

- Rename the "Resolution in this branch" column to "Resolution".
- State fixes with DREAM3D-NX release versions only.
- Add Git terms to the list of content that a report must not have.
  The Verified commit row and the provenance sidecar keep commit and PR
  evidence.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…vidence paragraph

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…roval

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
… the report template

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
… template

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…late

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…rt template

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…t template

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…rt template

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…s V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…umNeighbors V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…rison

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…deviations

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…ty V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…orCAxisMisalignments V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…ors V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…ids V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
… V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…uireMinNumNeighbors V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…Axis Orientations

- Check at execute time that every Feature Id indexes the Average C-Axes
  array (error -76405). Before, an out-of-range Feature Id wrote past the
  end of the feature cache.
- Accumulate the c-axis sums in double and convert to float32 once. Before,
  the float32 sums lost precision for large features.
- Add tests for out-of-range Feature Ids and an analytical large-feature
  case. Both fail on the old code.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…e C-Axis Misalignments

- Check at execute time that every Feature Id indexes the Average C-Axes
  array (error -9806). Before, an out-of-range or negative Feature Id read
  and wrote past the end of the feature arrays.
- Apply the same cell inclusion rule (positive Feature Id, positive
  hexagonal phase) in the standard deviation pass as in the mean pass.
  Before, excluded cells added zeros and inflated the standard deviation.
- Accumulate the feature means in double and convert to float32 once.
- Add tests for each defect. Each test fails on the old code.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
… Centroids from Triangle Geometry

- Preflight rejects an empty Face Labels array (-12903) and a Face Labels
  array whose tuple count is not the number of faces (-12902). Before,
  these inputs read out of bounds or requested a huge resize.
- Resize the feature matrix only for positive labels, with 64-bit
  comparison.
- Sum vertex coordinates in double and convert to float32 once.
- The exemplar comparison tolerance changes from 0.0001 to 0.05. The
  archive holds float32-accumulated centroids; a float64 reference
  computed from the archive triangles differs from them by at most
  0.0415 (feature 518). The archive is unchanged.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…ple)

The connected-component path marked every voxel of a 2D image as a
boundary voxel because it tested the unit axis, so no hole was filled.
The BFS path already ignored unit axes. The boundary test now ignores
an axis of size 1. Add bool and uint8 tests for all three 2D
orientations; they fail on the old code.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Describe when V&V work happens after a filter has a V&V record, so that
engineers and their tools apply one approach:

- Proposed trigger: a change that alters output for valid input needs a
  scoped V&V update in the same PR; other changes need only unchanged
  V&V tests; a release gate runs the legacy comparisons again.
- Agreed: all V&V work stays on one branch until the policy and the
  report template are stable, with code fixes in separate commits.
- Agreed: the three-way legacy comparison packages (stock DREAM.3D
  v6.5.171, a local proof version, DREAM3D-NX) and their archive rules.

Link the new file from vv_policy.md.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
@imikejackson
imikejackson force-pushed the topic/vv_report_cleanup branch from 40268f0 to 79962a8 Compare October 24, 2026 13:31
@imikejackson imikejackson changed the title DOC: Align the V&V reports with the V&V report template BUG: Fix defects in V&V'd filters and align their V&V records with the template Oct 24, 2026
…Check V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…isMisorientations V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…tions V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…tion V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…cord

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…thNeighborValues V&V record

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant