Matrix summation rounding error?

Dear all,

I'm having trouble with summation of matrices and difference of the two sum. Here some lines of code:

thetaMN_prev = cellfun(@nansum, theta_prev);
cost_prev = sum(thetaMN_prev,'all');

thetaMN_try = cellfun(@nansum, theta_try);
cost_try = sum(thetaMN_try,'all');

cost_prev - cost_try often returns 0, while

sum(thetaMN_try - thetaMN_prev,'all') ~= 0, but this should be the same calculation mathematically speaking.

When this happens, cost_try and cost_prev are of the order of 1e19.

Is this an error due to some rounding process in function sum(A,'all')?

[Stephen23]

"... this should be the same calculation mathematically speaking."

Sure, but computing with binary floating point numbers is not the same as mathematics that you learned at school.

In particular, floating point addition is not associative:

https://blogs.mathworks.com/cleve/2024/06/27/supersum-in-defense-of-floating-point-arithmetic/

Answer 1

[Harshavardhan]

Hi @Giuseppe Gallo

The issue you're experiencing is likely due to floating-point precision errors. When dealing with very large numbers (e.g., 1e19), small differences can be lost due to rounding errors inherent in floating-point arithmetic. This can occur when subtracting two nearly equal large numbers, leading to a loss of precision.

We see this difference in results due to the order of operations:

• Summing First, Then Subtracting:

cost_prev - cost_try

• Here you're summing all elements of "thetaMN_prev" and "thetaMN_try" separately and then subtracting the two sums. This approach can amplify precision errors, because the subtraction of two large numbers can lead to significant cancellation error.
• Subtracting Element-wise, Then Summing:

sum(thetaMN_try - thetaMN_prev, 'all')

• Here you're subtracting corresponding elements of "thetaMN_try" and "thetaMN_prev first", and then summing the resulting differences. This method can be more accurate because it avoids the large intermediate sums and directly computes the overall difference, reducing the impact of floating-point precision errors.

I suggest using the second approach as its generally more accurate and can help minimize precision errors.

Hope this helps.