diff options
| author | Adrien Hopkins <adrien.p.hopkins@gmail.com> | 2023-09-19 19:44:35 -0500 |
|---|---|---|
| committer | Adrien Hopkins <adrien.p.hopkins@gmail.com> | 2023-09-19 19:44:35 -0500 |
| commit | 3cf61177898fad4e6be85b52083e9c1b508d6568 (patch) | |
| tree | 8ba1c616a16d15a6b05fec44481a4afca22b977b /factors/factors_test.go | |
| parent | 165e55184e79553c74b9c9056fd46f1b37a2b5d9 (diff) | |
factors.Score: Avoid overflow by using math/big
When using really large numbers, factors.Score could overflow, which
would cause an incorrect result. Using arbitrary-precision arithmetic
fixes this. I only do so above 2^28, since below then factor sums are
guaranteed to not overflow, and normal arithmetic is faster.
Diffstat (limited to 'factors/factors_test.go')
| -rw-r--r-- | factors/factors_test.go | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/factors/factors_test.go b/factors/factors_test.go index a18d30b..838186f 100644 --- a/factors/factors_test.go +++ b/factors/factors_test.go @@ -94,6 +94,8 @@ var factorScoreCases = map[uint]float64{ 10: 1.8, 12: 7.0 / 3.0, 120: 3.0, + // number that will use bigScore + 367567200: 62496.0 / 12155.0, } func TestFactorScore(t *testing.T) { |