Post by bobpit on Jul 4, 2015 13:20:04 GMT
Hello
Consider the following code:
Questions:
1) How come r4 is 0.300000011920929 ? I would expect it to be 0.300000000000000
2) How can I get a pure 0.3 number and proceed to other calculations with more accuracy (like if I wanted to calculate 12.45 * 0.3).
3) %f always displays 6 decimal digits?
Consider the following code:
local x;
x=0.3;
printf( "r1 = %f\n", x );
printf( "r2 = %5.15f\n", x );
printf( "r3 = %f\n", prec(x,4) );
printf( "r4 = %5.15f\n", prec(x,4) );
x=10.12981;
printf( "r5 = %f\n", x);
printf( "r6 = %5.15f\n", x);
printf( "r7 = %f\n", prec( x, 2 ));
printf( "r8 = %5.15f\n", prec( x, 2 ));
printf( "r9 = %5.15f\n", round( x ));
printf( "r10 = %5.15f\n", round( x*100 )/100);
This is the output:r1 = 0.300000
r2 = 0.300000011920929
r3 = 0.300000
r4 = 0.300000011920929
r5 = 10.129810
r6 = 10.129810333251953
r7 = 10.120000
r8 = 10.119999885559082
r9 = 10.000000000000000
r10 = 10.130000114440918
Questions:
1) How come r4 is 0.300000011920929 ? I would expect it to be 0.300000000000000
2) How can I get a pure 0.3 number and proceed to other calculations with more accuracy (like if I wanted to calculate 12.45 * 0.3).
3) %f always displays 6 decimal digits?