Numeric value returns 16 decimal places

Tap Forms – Organizer Database App for Mac, iPhone, and iPad Forums Script Talk Numeric value returns 16 decimal places

Viewing 8 reply threads
  • Author
  • November 27, 2019 at 12:51 AM #38315


    I’ve a simple loop over all records.

    The table has several same fields (Number, Currency Style, 2 Decimal Places).

    Playing with that, every field has the value 1.99

    In some cases getFieldValue returns “1.9899999999999998”

    Is this a bug ?

    November 27, 2019 at 3:04 AM #38326


    It’s a byproduct of the use of the Objective-C doubleValue method. The formatting and rounding according to the decimal places and Number Format settings doesn’t happen until display time. But internally the value is as it is. You would need to perform some JavaScript rounding of the values yourself if you need to do that.

    November 27, 2019 at 8:34 AM #38332


    Thanks, Brendan.
    I tried that already by using var.toFixed(2),
    but this is running in a TypeError: toFixed is not a function.
    This happens only when the script is running over such an affected value, for other values toFixed is working.

    November 27, 2019 at 6:15 PM #38350

    Sam Moffatt

    My guess is that the number if actually a string, use typeof to check what type it is. If it’s a string then it wouldn’t have toFixed because that’s for numbers. If it is a string then perhaps try a parseFloat as a workaround.

    November 28, 2019 at 12:44 AM #38359


    But should this happen when the field is numeric ?
    However, found a workaround to fix it.

    November 28, 2019 at 9:51 AM #38373

    Sam Moffatt

    Probably shouldn’t happen but bugs happen :)

    What was your workaround?

    November 28, 2019 at 11:26 AM #38375


    Ugly and sort of tricky.
    Detecting the amount of decimal places.
    If > 2 split the string to x.xx and convert to number.

    Still unclear for me why this happens only for some fields and some values only, even if they all have the same format and the same value.

    November 28, 2019 at 2:45 PM #38382

    Dave Emme

    Sorry, but that’s life when using floating point (non-integer) numbers in a general-purpose computer. It’s not anybody’s bug. It’s down to the fact that the representation of arbitrary floating point numbers in a fixed number of bits in the hardware can result in the computer being unable to represent some values exactly. So, for example, a floating point calculation which should result in 2.000 might instead give 1.99999…

    December 27, 2019 at 9:39 PM #38386

    Sam Moffatt

    Yes, this is a standard problem with floating point numbers that values aren’t represented exactly. but are an approximation. Some values approximate well and others, as you’ve seen, end up looking a little weird if you don’t round them. There is an article called What Every Computer Scientist Should Know About Floating-Point Arithmetic which is perhaps a little heavy but is detailed on how floating point numbers are merely an approximation, the general standard is IEEE 754.

    Another workaround is to store everything as whole numbers and then elsewhere handle adjusting for display purposes. This allows you to work with whole numbers throughout though it requires at least two fields: one for the value and a second for number of significant digits. You could store it in a single value if and only if you have a fixed number of digits. Then it becomes more a formatting issue. The challenge with this approach is when you need to do maths with the numbers, you need to convert them to an equivalent formats and integer math is restricted in size in many CPUs which is why we have the floating point system we do. These days that is less of an issue, a 64-bit processor can store an unfathomably large number, however a 16-bit processors largest number is 65535, and half that if you want to store negative values.

    This is unfortunately less a Tap Forms issue and more an issue with hardware limitations of computers.

Viewing 8 reply threads

You must be logged in to reply to this topic.