Skip to content

Fix precision of bigdecimal.power(integer, prec)#342

Closed
tompng wants to merge 1 commit into
ruby:masterfrom
tompng:power_by_int_prec_fix
Closed

Fix precision of bigdecimal.power(integer, prec)#342
tompng wants to merge 1 commit into
ruby:masterfrom
tompng:power_by_int_prec_fix

Conversation

@tompng

@tompng tompng commented Jun 5, 2025

Copy link
Copy Markdown
Member

Fixes #352

Fixes this lack of precision bug

BigDecimal('0.1234').power(8, 12) #=> 0.53768e-7
BigDecimal('12.34').power(-8, 12) #=> 0.186e-8
BigDecimal('0.1234').power(8, 12)
# before: 0.53768e-7
# after:  0.537676775108e-7

BigDecimal('12.34').power(-8, 12)
# before: 0.186e-8
# after:  0.185985344038e-8

Too high precision gets lower in some case

BigDecimal('12.34').power(8.000000000001, 12)
# =>      0.537676775109e9 (doesn't change in this pull request)

BigDecimal('12.34').power(8, 12)
# before: 0.537676775108238564041e9 (precision too high)
# after:  0.537676775108e9 (precision consistent with power by non-integer)

@mrkn mrkn added this to the v3.3 milestone Jun 17, 2025
@mrkn mrkn force-pushed the power_by_int_prec_fix branch from 8af311d to b8d2982 Compare June 17, 2025 14:48
@tompng

tompng commented Jun 26, 2025

Copy link
Copy Markdown
Member Author

Don't need to review or merge if we're doing #347 soon.

Fixes this lack of precision bug
BigDecimal('0.1234').power(8, 12) #=> 0.53768e-7
BigDecimal('12.34').power(-8, 12) #=> 0.186e-8
@tompng tompng force-pushed the power_by_int_prec_fix branch from b8d2982 to 48066a9 Compare July 6, 2025 05:38
@tompng tompng closed this Jul 19, 2025
@tompng tompng deleted the power_by_int_prec_fix branch September 17, 2025 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Power low precision

2 participants