Error when using sptensor in parfor

I am trying to edit a sparse tensor from the Tensor Toolbox inside a parfor loop. However I am getting the "Unable to classify the variable 'tau' in the body of the parfor-loop. I thought this might be a slicing issue, however when I tried testing using a 3D matrix with smaller dimensions the parfor loop runs. Unfortunately the needed tensor size is too big for me to use a native 3D matrix, and so I am trying to stay with a sparse tensor if possible. Any ideas how I can correct this? A simplified code is provided:

tau = sptensor([],[],[N N M]);
opts = parforOptions(gcp,'RangePartitionMethod','fixed','SubrangeSize',100);
parfor (i = 1:N-1, opts)
    mytemp_p_k = sptensor([],[],[N M]);
    mytemp_p_k(a,b) = <edit matrix>
    tau(i,:,:) = mytemp_p_k;
end

[Matt J]

However I am getting the "Unable to classify the variable 'tau' in the body of the parfor-loop.

Do you mean a Code Analyzer warning? I, for one, get no such warning.

Answer 1

[Matt J]

You could try with ndSparse() instead of sptensor()

https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays

The following modified code ran fine for me.

[N,M]=deal(100);

tau = ndSparse.spalloc([N N M],N^2*M/100);


opts = parforOptions(gcp,'RangePartitionMethod','fixed','SubrangeSize',100);
parfor (i = 1:N-1, opts)
    mytemp_p_k = sprand(N,M,0.01);
    tau(i,:,:) = mytemp_p_k;
end

[Ezra Altabet]

Hi Matt,

Thank you for your help. With your code, when I try initializing tau with N = 1000189 and M = 343 (which are the values my code needs) I get the following error: "Sparse matrix sizes must be non-negative integers less than MAXSIZE as defined by COMPUTER." I didn't have this issue with sptensor, so I'm wondering if a settings thing I can change. Any ideas as to how I can fix this?

[Matt J]

The error should go away if you reshape your matrix as NxMxN and loop across the 3rd dimension instead of the first. Looping across the first dimension of tau is a less favorable memory access pattern anyway (probably for sptensor as well).

[N,M]=deal(1000189, 343);

tau = ndSparse.spalloc([N, M, N],  2e6);


opts = parforOptions(gcp,'RangePartitionMethod','fixed','SubrangeSize',100);
parfor (k = 1:N-1, opts)
    mytemp_p_k =...;
    tau(:,:,k) = mytemp_p_k;
end

I should also mention that is very inefficient to fill a sparse array cumulatively in a loop, as you are doing currently. This is true for any sparse type implementation. It would be much better if you used the loop to accumulate only the tabular I,J,K,S data needed to populate the array. Then at the end, you would build the actual sparse array:

[I,J,K,S]=deal(cell(N-1,1));

opts = parforOptions(gcp,'RangePartitionMethod','fixed','SubrangeSize',100);
parfor (k = 1:N-1, opts)
    
    mytemp_p_k =...;
       
    
    [I{k},J{k},S{k}]=find(mytemp_p_k);
    
    K{k}=repmat(k,size(S{k}));

end

I=cell2mat(I(:));
J=cell2mat(J(:));
K=cell2mat(K(:));
S=cell2mat(S(:))

tau=ndSparse.build([I,J,K],S);

[Ezra Altabet]

Thank you for your help! I'm trying to implement your method and I appear to be getting some discrepencies in my data. I would think this would produce an NxNxM matrix but instead it outputs an NxMxN matrix. Further, as a test, when I saved mytemp_p_k into a cell after each iteration I got an NxM matrix. Normally this wouldn't be concerning, but ti looks like I'm getting different data then when using my test code that jut saves the value of each sparse matrix as a new cell. Any help would be appreciated. This is the code that I am using:

[I,J,K,S]=deal(cell(N-1,1));
mycell = {};

for i = 1:N
    mycell{i} = 0; % preallocate number of cells
end

parfor (i = 1:N-1)
    mytemp_p_k = sparse(N, M);
    mytemp_p_k(a,b) = <edit matrix>
    mycell{i} = mytemp_p_k;
    [I{i},J{i},S{i}]=find(mytemp_p_k);
    K{i}=repmat(i,size(S{i}));
end
I=cell2mat(I(:));
J=cell2mat(J(:));
K=cell2mat(K(:));
S=cell2mat(S(:));
tau=ndSparse.build([I,J,K],S);

[Matt J]

Correct, we want it to be an NxMxN matrix to avoid the MAXSIZE error you discovered above, and also because acessing tau(:,:,i) is more efficient than accessing tau(i,:,:) due to memory locality.

[Ezra Altabet]

Thank you. Just out of curiosity, is there a way that I could use just the tabular data format so that I can write to any element in the parfor loop instead of just a slice, or does that violate the limits of parfor? In my specific example, elements that need to be changed only get set to 1, if that simplifies the problem.

[Matt J]

I don't follow.