calculation error for floor function

조회 수: 9 (최근 30일)
Nishanth Kumar
Nishanth Kumar 2017년 2월 6일
댓글: Shlomo Geva 2017년 10월 29일
Hello all, when I was trying for >>floor(2.3 * 50) , I was getting 114 where I suppose to get 115. Is this due to precision issue? and How to fix it?
  댓글 수: 1
Shlomo Geva
Shlomo Geva 2017년 10월 29일
Well, check this code: floor(p/65536) floor() does not handle correctly uint32 values of p > 4294934528 which is well within uint32 range. It is not even returning a value in the expected range 0 to 65535
>> p=uint32([4294934527, 4294934528, 2^32-1]);floor(p/65536)
ans =
1×3 uint32 row vector
65535 65536 65536

댓글을 달려면 로그인하십시오.

답변 (2개)

Sebastian
Sebastian 2017년 2월 6일
If you have the Symbolic Math Toolbox, try this:
x = vpa(2.3 * 50)
x =
115.0
x = floor(x)
x =
115

Jan
Jan 2017년 2월 6일
Yes, this is an effect of the limited precision of the IEEE754 floating point standard. See http://matlab.wikia.com/wiki/FAQ#Why_is_0.3_-_0.2_-_0.1_.28or_similar.29_not_equal_to_zero.3F . You find a lot of corresponding question, when you search e.g. for "faq 6.1".

카테고리

Help CenterFile Exchange에서 Numbers and Precision에 대해 자세히 알아보기

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by