Numeric value returns 16 decimal places

Tap Forms – Organizer Database App for Mac, iPhone, and iPad Forums Script Talk Numeric value returns 16 decimal places

Viewing 7 reply threads
  • Author
    Posts
  • November 27, 2019 at 12:51 AM #38315

    Marcus
    Participant

    Hi,
    I’ve a simple loop over all records.

    The table has several same fields (Number, Currency Style, 2 Decimal Places).

    Playing with that, every field has the value 1.99

    In some cases getFieldValue returns “1.9899999999999998”

    Is this a bug ?

    November 27, 2019 at 3:04 AM #38326

    Brendan
    Keymaster

    It’s a byproduct of the use of the Objective-C doubleValue method. The formatting and rounding according to the decimal places and Number Format settings doesn’t happen until display time. But internally the value is as it is. You would need to perform some JavaScript rounding of the values yourself if you need to do that.

    November 27, 2019 at 8:34 AM #38332

    Marcus
    Participant

    Thanks, Brendan.
    I tried that already by using var.toFixed(2),
    but this is running in a TypeError: toFixed is not a function.
    This happens only when the script is running over such an affected value, for other values toFixed is working.

    November 27, 2019 at 6:15 PM #38350

    Sam Moffatt
    Participant

    My guess is that the number if actually a string, use typeof to check what type it is. If it’s a string then it wouldn’t have toFixed because that’s for numbers. If it is a string then perhaps try a parseFloat as a workaround.

    November 28, 2019 at 12:44 AM #38359

    Marcus
    Participant

    But should this happen when the field is numeric ?
    However, found a workaround to fix it.

    November 28, 2019 at 9:51 AM #38373

    Sam Moffatt
    Participant

    Probably shouldn’t happen but bugs happen :)

    What was your workaround?

    November 28, 2019 at 11:26 AM #38375

    Marcus
    Participant

    Ugly and sort of tricky.
    Detecting the amount of decimal places.
    If > 2 split the string to x.xx and convert to number.

    Still unclear for me why this happens only for some fields and some values only, even if they all have the same format and the same value.

    November 28, 2019 at 2:45 PM #38382

    Dave Emme
    Participant

    Sorry, but that’s life when using floating point (non-integer) numbers in a general-purpose computer. It’s not anybody’s bug. It’s down to the fact that the representation of arbitrary floating point numbers in a fixed number of bits in the hardware can result in the computer being unable to represent some values exactly. So, for example, a floating point calculation which should result in 2.000 might instead give 1.99999…

Viewing 7 reply threads

You must be logged in to reply to this topic.