Very easy issue with division
조회 수: 1 (최근 30일)
이전 댓글 표시
I'm sure this is something very simple, but I've been messing around and can't figure it out...
I read in a value and store it as a 1x1 int32. Say the variable is resolution and its value is 100,000.
resolution/1e6
ans=0
No. But it I do:
x=100000;
x/1e6
and=0.1
Which is correct. Something is odd and I can't figure it out.
댓글 수: 0
답변 (1개)
bym
2011년 11월 10일
do a
whos(resolution)
whos(x)
You will find that x is a double and resolution is an int32. Since resolution/1e6 is cast as an int32 is comes out 0
댓글 수: 0
참고 항목
카테고리
Help Center 및 File Exchange에서 Graphics Performance에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!