Skip to content

Harden the GFS ingest (retries, missing-value guards, write order) [skip release]#17

Merged
johncarmack1984 merged 1 commit into
mainfrom
audit/rust-hardening
Jun 27, 2026
Merged

Harden the GFS ingest (retries, missing-value guards, write order) [skip release]#17
johncarmack1984 merged 1 commit into
mainfrom
audit/rust-hardening

Conversation

@johncarmack1984

Copy link
Copy Markdown
Owner

From the audit (rust-correctness lane). Backend robustness only — no data-format or user-facing change, hence [skip release].

  • Retries: fetch_field retries transient failures (3 attempts, backoff). One flaky GET in the ~57-step planet-wide fan-out no longer aborts the whole refresh. (audit: low — "no retry on the GFS fan-out")
  • Missing-value guard: a non-finite (missing) wind cell encodes to the mid-byte (~0 m/s) instead of saturating to byte 0, which the web denormalized to a false −40 m/s gale; scalar textures map non-finite to the transparent floor. (audit: low — the headline latent trap)
  • idx parsing: distinguishes "no next record" (EOF) from a present-but-unparseable next offset (corruption, surfaced), and guards the end-1 underflow with checked_sub. (audit: low)
  • Write ordering: fetch_temps writes lattice.json before the citytile/latest.json pointer, so the pointer commits last and a reader never races a half-written snapshot. (audit: low)
  • NWS_AREA validated as comma-separated 2-letter codes before going into the alerts query string. (audit: low)

fmt, clippy -D warnings, and the gfs/contract/s3 tests all pass.

Docs & attribution

  • N/A — backend hardening, no source/behavior/cost change.

…ring

- fetch_field retries transient failures (3 attempts, backoff) so one flaky GET
  in the ~57-step planet-wide fan-out no longer aborts the whole refresh.
- A non-finite (missing) wind cell now encodes to the mid-byte (~0 m/s) instead
  of saturating to byte 0, which the web read as a false -40 m/s gale; scalar
  textures map non-finite to the transparent floor.
- idx parsing distinguishes "no next record" (EOF, read to end) from a present-
  but-unparseable next offset (corruption, surfaced) and guards the end-1
  underflow with checked_sub.
- fetch_temps writes lattice.json before the citytile/latest.json pointer, so
  the pointer commits last and a reader never races a half-written snapshot.
- NWS_AREA is validated as comma-separated 2-letter codes before it goes into
  the alerts query string.
@johncarmack1984 johncarmack1984 merged commit 549f55a into main Jun 27, 2026
7 checks passed
@johncarmack1984 johncarmack1984 deleted the audit/rust-hardening branch June 27, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant