Skip to content

Floor and clamp the exponent of a root of zero - #153

Open
youdie006 wants to merge 1 commit into
cockroachdb:masterfrom
youdie006:root-zero-exponent
Open

youdie006 wants to merge 1 commit into
cockroachdb:masterfrom
youdie006:root-zero-exponent

Conversation

@youdie006

@youdie006 youdie006 commented Sep 11, 2026 •

Copy link
Copy Markdown

Sqrt and Cbrt of a zero return the wrong exponent and never clamp.

c := apd.BaseContext.WithPrecision(10)
x, _, _ := apd.NewFromString("0E-9")
c.Sqrt(d, x)   // 0.0000  (exponent -4)

The repo's own vendored vector disagrees — testdata/squareroot.decTest:3780 says sqtx9010 squareroot 0E-9 -> 0.00000, exponent −5, and Python's decimal agrees. Every odd negative exponent is off by one. The clamp is missing outright: with MaxExponent: 9, Sqrt of 0E+100 returns exponent 50 and no Clamped.

rootSpecials (context.go:462-465) does d.Exponent /= factor, and Go's / truncates toward zero where the ideal exponent is floor(e/factor); the result is then never brought into [etiny, MaxExponent] nor flagged. The sibling quoSpecials (context.go:247-250) already does both. I used c.etiny() and c.goError rather than setExponent, because setExponent also raises Rounded, which these vectors do not expect.

Why 22,915 conformance cases never caught it: gda_test.go:714-715 compares finite results with d.Cmp(r), which is numeric, so every zero compares equal to every other zero regardless of exponent. The decTest files carry the exponents; the comparison discards them.

One thing to decide: Cbrt is collateral

rootSpecials is shared, so this changes Cbrt too, and testdata/cuberoot-apd.decTest — this repo's own hand-written file — is already inconsistent about zeros:

cbtx006 cuberoot 00.0  ->  0      (operand exponent -1, expected exponent  0)
cbtx016 cuberoot -00.0 -> -0.0    (operand exponent -1, expected exponent -1)

Same operand exponent, contradictory expectations; cbtx022 cuberoot 0E+5 -> 0 also disagrees with cbtx017/cbtx020. None of it is enforced, for the same Cmp reason. I did not touch that data. Happy to correct those lines in a follow-up, or to restrict this to factor == 2 and leave Cbrt alone — your call.

Verification
row result
new test on master FAIL at sqtx006
with the fix PASS
floor unconditionally (drop the x.Exponent < 0 test) FAIL at sqtx9037
clamp the value but drop the Clamped flag FAIL at sqtx9024, got conditions "", expected "clamped"

This lets 7 entries out of GDAignoreFlags, checked both ways:

  • fix applied, the 7 removed: TestGDA exits 0.
  • library left pristine, the 7 removed: TestGDA fails on exactly those 7 IDs. So the removals are earned by the fix, not by loosening the test.
  • sqtx9045 stays: removing it too still fails, because apd does not implement the clamp: directive. I corrected its comment to say so.

CI rows on go1.26.3: go test ./..., -race, -bench=. -benchtime=1x, go vet -unsafeptr=false ./..., staticcheck ./..., and go test -c for GOARCH=arm GOARM=7 and arm64 — all exit 0. I did not run a real go1.17/1.19 toolchain; the diff uses only /, % and existing identifiers.

gofmt -l . flags four files on pristine too (go1.26's gofmt reformats the //gcassert: comments), so I did not run gofmt -w; the diff contains only the rootSpecials hunk.


Disclosure: I used Claude (an AI assistant) while preparing this change. Every result above I ran and verified myself.

rootSpecials divided the operand exponent by factor with Go's /, which
truncates toward zero, so every odd negative exponent came out one too
high. It also never brought the result into the context's exponent range
and never raised Clamped, unlike quoSpecials.

TestGDA compares finite results with Cmp, which treats every zero as
equal regardless of exponent, so the vendored vectors never caught it.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-11T06:33:33.372129Z 06d2103 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06d21033f3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread context.go
Comment on lines +467 to +468
if x.Exponent%factor != 0 && x.Exponent < 0 {
e--

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve Cbrt's zero exponent contract

Because rootSpecials is also called by Cbrt, flooring here changes positive cube-root results with negative, non-divisible exponents: for example, Cbrt("00.0") now returns 0.0 with exponent -1, while testdata/cuberoot-apd.decTest cases cbtx006–cbtx008 require exponent 0. TestGDA does not expose this regression because it compares zeros numerically, and the new targeted test only selects the contradictory negative-zero expectations. Restrict the flooring change to square root, or first define and consistently update and test the cube-root contract.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant