-
Notifications
You must be signed in to change notification settings - Fork 7
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
vectors and quaternions currently use %.6g for formatting their components, which leads to loss of significant precision when converting to strings:
> v = vector(123456.9,0,0)
> v
<123457, 0, 0>
> v.x
123456.8984375
Simulating the difference between %.6g and %.6f in Python:
>>> "%.6g" % 123456.9
'123457'
>>> "%.6f" % 123456.9
'123456.900000'showing that %.6g is clearly not the correct way to go here. The number apparently refers to how many significant digits we want in the output.
Since our main goal is to get rid of non-significant zeros to the right of the decimal place, we should probably just use %f plus manual truncation of uninteresting zero padding. There's probably existing float formatting in the Luau code that might be useful as well.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working