How do I fix my output for floating-point imprecision? - actionscript-3

Question

I am doing some float manipulation and end up with the following numbers:
-0.5
-0.4
-0.3000000000000000004
-0.2000000000000000004
-0.1000000000000000003
1.10E-16
0.1
0.2
0.30000000000000000004
0.4
0.5
The algorithm is the following:
var inc:Number = nextMultiple(min, stepSize);
trace(String(inc));
private function nextMultiple(x:Number, y:Number) {
return Math.ceil(x/y)*y;
}
I understand the fact the float cannot always be represented accurately in a byte. e.g 1/3. I also know my stepsize being 0.1. If I have the stepsize how could I get a proper output?
The strange thing is that its the first time I've encountered this type of problem.
Maybe I dont play with float enough.

Answer 1

A language agnostic solution would be to store your numbers as an integer number of steps, given that you know your step size, instead of as floats.
A non-language agnostic solution would be to find out what your language's implementation of printf is.
printf ("float: %.1f\n", number);

Answer 2

The limited floating point precision of binary numbers is your problem, as you recognize. One way around this is not to do floating point math. Translate the problem to integers, then translate back for the output.

Answer 3

Either use integers instead of a floating point type, or use a floating point type where the "point" is a decimal point (e.g. System.Decimal in .NET).

Answer 4

If you're using a language with a round function, you can use that.
Edit
In response to comments about rounding, here's a sample in c#:
float value = 1.0F;
for (int i = 0; i < 20; i++)
{
value -= 0.1F;
Console.WriteLine(Math.Round(value, 1).ToString() + " : " + value.ToString());
}
The results are:
0.9 : 0.9
0.8 : 0.8
0.7 : 0.6999999
0.6 : 0.5999999
(etc)
The rounding does resolve the precision problem. I'm not arguing that it's better than doing integer math and then dividing by 10, just that it works.

Answer 5

With your specific problem, count from -5 to 5 and divide by 10 before actually using the value for something.

Answer 6

I did the following,
var digitsNbr:Number = Math.abs(Math.ceil(((Math.log(stepSize) / Math.log(10))) + 1));
tickTxt.text = String(inc.toPrecision(digitsNbr));
Its not efficient but i dont have many steps.
======
I should just get the nbr of steps as an int and multiply by step ...

Answer 7

If you don't have printf, or if the steps are not just powers of 10 (e.g. if you want to round to the nearest 0.2) then it sounds like you want a quantizer:
q(x,u) = u*floor(x/u + 0.5);
"u" is the step size (0.1 in your case), floor() finds the greatest integer not greater than its input, and the "+ 0.5" is to round to the nearest integer.
So basically, you divide by the step size, round to the nearest integer, and multiply by the step size.
edit: oh, never mind, you're basically doing that anyway & the step where it's multiplying by u is introducing rounding error.

Answer 8

Simply scale the numbers to obtain integers then do maths and scale them back to floats for display:
//this will round to 3 decimal places
var NUM_SCALE = 1000;
function scaleUpNumber(n) {
return (Math.floor(n * NUM_SCALE));
}
function scaleDnNumber(n) {
return (n / NUM_SCALE);
}
var answ = scaleUpNumber(2.1) - scaleUpNumber(3.001);
alert(scaleDnNumber(answ)); // displays: -0.901
Change NUM_SCALE to increase/decrease decimap places
|/|ax

Answer 9

Your best bet is to use a Decimal data type if your language supports it. Decimals were added to a number of languages to combat this exact problem.

Answer 10

This is a bit counter-intuitive, but I tested it and it works (example in AS3):
var inc:Number = nextMultiple(min, stepSize);
trace(String(inc));
private function nextMultiple(x:Number, y:Number) {
return Math.ceil(x/y)*(y*10)/10;
}
So the only thing I added is multiplying y by 10, then dividing by 10. Not an universal solution but works with your stepSize.
[edit:] The logic here seems to be that you multiply by a big enough number so as for the last decimal digits to "drop off the scale", then divide again to get a rounded number. That said, the example above which uses Math.round() is more readable and better in the sense that the code explicitly says what will happen to the numbers passed in.