Nested parfor loop slows down with each iteration

I have the following (simplified) code example with a nested outer parfor and inner for loop. Variables are sliced. It takes about 30 minutes for the first 10 outer iterations (running on 10 workers). The second round (iterations 10-20) takes more than an hour and it keeps increasing for later iterations. What is wrong here?

function my_function()

n_iterations_v = cell(1,50);
n_iterations_v = init_iterations(); % each element is an integer
parameters_v = cell(1,50); 
parameters_v = init_params() % element i is a cell array with n_iterations_v{i} elements (where each element is a struct)

% outer loop
parfor (k = 1:50)
    results_v = cell(1,n_iterations_v{k});
    
    % inner loop
    for i=1:n_iterations_v{k}
        results_v{i} = some_other_function(parameters_v{k}{i});
    end
      
    save(filename, "-fromstruct", struct("results_v",results_v));
end

end

Answer 1

[Jasvin]

Have a couple of suggestions on how to investigate further.

Can you measure whether the same behaviour is observed for a regular "for" loop as well? If that's the case then it would imply that this is expected behaviour and parallelization can only do so much and the performance has to do more with the business logic of your code instead.

I also found this StackOverflow answer that suggests to use "parpool('threads')" which could be applicable in your case since you too seem to have a decent amount of data along with the copy operation involved.

https://stackoverflow.com/a/69955780

You could also create a technical support case with the MATLAB team by contacting [e-mail]