mle memory error with custom negative log-likelihood function

I am trying to use 'mle' with a custom negative log-likelihood function, but I get the following error:

Requested 1200000x1200000 (10728.8GB) array exceeds maximum array size preference (15.6GB). This might cause MATLAB to become unresponsive.

The data I am usng is a 1x1200000 binary array (which I had to convert to double), and the function has 10 arguments: one for the data, 3 known paramenters, and 6 to be optimized. I tried setting 'OptimFun' to both 'fminsearch' and 'fmincon'. Also, optimizing the parameters using 'fminsearch' and 'fminunc' instead of 'mle' works fine.

The problem happens in the 'checkFunErrs' functions, inside the 'mlecustom.m' file (call at line 173, actuall error at line 705).

I am quite new to MATLAB, so I am guessing I do not understand how to use this properly, or doesn't mle support that ammount of data?

Thanks for the help.

data = randi([0 1], 1, 1200000);

T_1 = 50000;

T_2 = 100000;

npast = 10000;

start = [0 0 0 0 0 0];

func = @(x, data, cens, freq)loglike(data, [x(1) x(2) x(3) x(4) x(5) x(6)],...

T_1, T_2, npast);

params = mle(data, 'nloglf', func, 'Start', start, 'OptimFun', 'fmincon');

% Computes the negative log likehood

function out = loglike(data, params, T_1, T_2, npast)

size = length(data);

if npast == 0

past = 0;

else

past = zeros(1, size);

past(npast+1:end) = movmean(data(npast:end-1),[npast-1, 0]); % Average number of events in the previous n years

end

lambda = params(1) + ...

(params(2)*cos(2*pi*(1:size)/T_1)) + ...

(params(3)*sin(2*pi*(1:size)/T_1)) + ...

(params(4)*cos(2*pi*(1:size)/T_2)) + ...

(params(5)*sin(2*pi*(1:size)/T_2)) + ...

params(6)*past;

out = sum(log(1+exp(lambda))-data.*lambda);

end

Answer 1

[Mihaela Jarema]

Hi Jose,

The error you mention is thrown when mle calls your custom function loglike, in the last line of the custom function loglike:

out = sum(log(1+exp(lambda))-data.*lambda);

It’s the data.*lambda part which throws the error, since

• data is passed as a 1200000 x 1 column vector
• you create lambda as a 1 x 1200000 row vector
• data.*lambda will then do implicit expansion to 1200000 x 1200000 because the two arrays have compatible sizes (either the same or 1, in this case 1), repeating the columns in data and the rows in lambda and multiplying the two matrices element-wise.
• The implicit expansion fails because MATLAB has built-in protection against creating arrays that are too large.

To avoid implicit expansion, you might want to try out working with column vectors instead of row vectors, e.g., by changing the order of the arguments in the first line from 1, 1200000 to 1200000,1 and transposing others like 1:size to (1:size)’. By the way, you might also want to rename the variable size to something else, like len, since size is a built-in MATLAB function you are overwriting.

Otherwise, you could try some of the recommendations to resolve out of memory errors.

Hope this helps,

Mihaela