fix(test): remove non-deterministic tiebreaks in otel snapshot ordering

[aaronArinder]

Summary

• test_batch_trace_id failed intermittently in CI (insta snapshot mismatch, confirmed on amd_linux_test 2026-07-09) due to sort_spans_for_snapshot not fully canonicalizing span order.
• Two independent, compounding bugs, both matching the "non-deterministic collection ordering in assertions" pattern:
  1. Root ordering leaked HashMap iteration order. Roots were collected via spans_by_root.keys().cloned().collect(). Rust's default hasher is randomly seeded per process, so whenever two roots tied on (start_time, end_time, name) — which test_batch_trace_id hits routinely, since its two batched operations produce two structurally-identical supergraph root spans that can start/end within the same nanosecond — the pre-sort order (and therefore the stable-sort tiebreak outcome) depended on hash-iteration order, not content.
  2. Sibling tiebreak used raw arrival order. The final tiebreak was original_position_within_parent — exactly the tokio-scheduling arrival sequence the function exists to cancel out.
• Confirmed this is not a single-flush issue: get_traces already polls up to 10s for a report matching a shape filter, not reports[0], so that part was already handled correctly.

Fix

• Kept (start_time_unix_nano, end_time_unix_nano) as the primary sort key, so non-tied siblings (the overwhelming common case) retain today's order and existing snapshots don't need to be regenerated.
• Replaced only the tiebreak with a bottom-up content signature (name + own attributes with volatile keys stripped + children's signatures), instead of anything positional.
• Root/child lookups are now a direct index scan + keyed lookup — no more wholesale HashMap iteration influencing order.
• Unified the volatile-attribute list into a single VOLATILE_ATTRIBUTE_KEYS const shared with assert_report!'s redaction list, closing a drift risk between "redacted for display" and "excluded from sort key."

An earlier draft made the entire key content-based (no timestamps). That's a valid total order but reorders every non-tied sibling too, churning nearly every snapshot in the file — rejected in favor of the hybrid above.

Test plan

• test_batch_trace_id alone: 40/40 passes
• Full apollo_otel_traces suite (12 tests): 12/12 full-suite runs (144 individual executions), via cargo nextest per this file's serialization requirement
• cargo clippy --test apollo_otel_traces and cargo fmt --check clean
• No .snap files needed regeneration — the hybrid design preserves existing checked-in snapshots exactly

Test-only change; no production code touched.

🤖 Generated with Claude Code

[apollo-librarian]

✅ Docs preview has no changes

The preview was not built because there were no changes.

Build ID: 5addf779add524081cb1dd4c
Build Logs: View logs

---

✅ AI Style Review — No Changes Detected

No MDX files were changed in this pull request.

Review Log: View detailed log

This review is AI-generated. Please use common sense when accepting these suggestions, as they may not always be accurate or appropriate for your specific context.

[github-actions]

@aaronArinder, please consider creating a changeset entry in /.changesets/. These instructions describe the process and tooling.