calculation error for floor function
조회 수: 9 (최근 30일)
이전 댓글 표시
Hello all, when I was trying for >>floor(2.3 * 50) , I was getting 114 where I suppose to get 115. Is this due to precision issue? and How to fix it?
댓글 수: 1
Shlomo Geva
2017년 10월 29일
Well, check this code: floor(p/65536) floor() does not handle correctly uint32 values of p > 4294934528 which is well within uint32 range. It is not even returning a value in the expected range 0 to 65535
>> p=uint32([4294934527, 4294934528, 2^32-1]);floor(p/65536)
ans =
1×3 uint32 row vector
65535 65536 65536
답변 (2개)
Sebastian
2017년 2월 6일
If you have the Symbolic Math Toolbox, try this:
x = vpa(2.3 * 50)
x =
115.0
x = floor(x)
x =
115
댓글 수: 0
Jan
2017년 2월 6일
Yes, this is an effect of the limited precision of the IEEE754 floating point standard. See http://matlab.wikia.com/wiki/FAQ#Why_is_0.3_-_0.2_-_0.1_.28or_similar.29_not_equal_to_zero.3F . You find a lot of corresponding question, when you search e.g. for "faq 6.1".
댓글 수: 0
참고 항목
카테고리
Help Center 및 File Exchange에서 Numbers and Precision에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!