Skip to content

Fix quantisation with constrained exponent limits - #157

Open
ryanduguid wants to merge 1 commit into
cockroachdb:masterfrom
ryanduguid:fix/quantize-temporary-exponent-limits
Open

ryanduguid wants to merge 1 commit into
cockroachdb:masterfrom
ryanduguid:fix/quantize-temporary-exponent-limits

Conversation

@ryanduguid

@ryanduguid ryanduguid commented Oct 1, 2026 •

Copy link
Copy Markdown

Summary

Fixes #94: quantising 6410.000000000000000000 to eight decimal places can return NaN under a valid currency context. The coefficient-rounding step uses a scaled temporary value whose exponent differs from the result. Apply system exponent limits to that temporary context, then validate the restored result under the caller's limits.

scaled coefficient -> temporary system limits
restored amount    -> caller's precision and exponent limits

The same helper also restores correct integral rounding in constrained contexts.

Evidence

  • Before: 396 of 546 focused leaf subcases fail on master, including the reported amount and ordinary amounts rounded to cents.
  • After: All 546 pass, checking exact coefficients, exponents, signs, flags, traps and result aliases. All 1,728 comparisons with independent Python decimal expectations agree; master matches 828.
  • The configured build, test, race, benchmark smoke and vet commands pass on Linux amd64 across Go 1.19.13, 1.20.14, 1.21.13, 1.22.12, 1.23.12 and 1.24.13. Applicable staticcheck and gcassert checks pass.
  • Each full suite reports 24,355 passing and 3,574 skipped test nodes. These counts include parent nodes.
  • Full tests and benchmark smoke pass under ARMv7 and AArch64 emulation with Go 1.24.0 binaries.

Merge danger

Door: two-way. Blast radius: arithmetic.

Previously rejected valid amounts now return finite results and their proper rounding flags. Genuine precision and exponent violations still return InvalidOperation, with trap behaviour covered by regressions.

Unverified

Upstream CI requires maintainer approval; the Go workflow has no jobs. Native ARM hardware, the declared Go 1.17 runtime and timing equivalence have not been tested. Benchmark runs are smoke checks. Existing system-boundary quantisation counterexamples remain; original and patched receipts agree. PR #152 addresses the separate directed-rounding failure when retained precision is negative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to use Quantize with limited precision context

1 participant