why does matlab distance function returns odd errors for specific doubles, how to fix?

Matlab distance function found here: /usr/local/apps/matlab-2013a/toolbox/map/map/distance.m generates an error when I give it certain values.

one example is that it would not work for these

    [hex2num('bff302fba03a0023'),hex2num('4003ff6217b27e6b'),hex2num('3ff302fba03a0023'),hex2num('bfe48a64f246bab5')]

    ans =
    -1.1882    2.4997    1.1882   -0.6419

when plugged in as so:

    distance(hex2num('bff302fba03a0023'),hex2num('4003ff6217b27e6b'),hex2num('3ff302fba03a0023'),hex2num('bfe48a64f246bab5'),'radians')

I get this error:

Error using atan2 Inputs must be real.

Error in distance>greatcircledist (line 160) rng = r * 2 * atan2(sqrt(a),sqrt(1 - a));

Error in distance (line 101) rng = greatcircledist(lat1, lon1, lat2, lon2, ellipsoid(1));

however rounding the values to

    distance(-1.1882   , 2.4997 ,   1.1882 ,  -0.6419,'radians')

works fine and returns 3.1416

[Chad Greene]

Interesting question. I am able to replicate the problem, but I don't know the cause.

Answer 1

[Steven Lord]

See Bug Report 946782. The way one of the quantities named a is computed makes it just slightly greater than 1 (by eps in this case) and when distance attempts to use sqrt(1-a) as one of the inputs to atan2 that makes that input complex. The atan2 function expects its inputs to be real and throws an error when that expectation is not met.

The reason rounding your values avoids this issue is because then the quantity a is just slightly smaller and that makes 1-a a very small positive number and so sqrt(1-a) is real.

[Pedro Ribeiro]

thank you! exactly what I needed