Skip to content

Improve precision of nested components in vectors and quaternions #58

@HaroldCindy

Description

@HaroldCindy

vectors and quaternions currently use %.6g for formatting their components, which leads to loss of significant precision when converting to strings:

> v = vector(123456.9,0,0)
> v
<123457, 0, 0>
> v.x
123456.8984375

Simulating the difference between %.6g and %.6f in Python:

>>> "%.6g" % 123456.9
'123457'
>>> "%.6f" % 123456.9
'123456.900000'

showing that %.6g is clearly not the correct way to go here. The number apparently refers to how many significant digits we want in the output.

Since our main goal is to get rid of non-significant zeros to the right of the decimal place, we should probably just use %f plus manual truncation of uninteresting zero padding. There's probably existing float formatting in the Luau code that might be useful as well.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions