accelerating a code via parfor

Is there any ways to speed up the code below? It is too slow on my computer? When I type in parpool, is that enough to use all the cores of my i5 CPU for calculating this code or must I insert a parfor code into my code below. If so, how do I insert a parfor code in the code below? Thanks

for x = 1:100
    for y =  1:61
             b = abs(bsxfun(@minus,q{x,y},kevin))
             averagesAcrossColumns = mean(b, 2);
             b = abs(bsxfun(@minus,b,averagesAcrossColumns));
             b = sum(b, 2);
             s{x,y} = b;
      end
  end

[Henrik]

Can you give some more information? What are the dimensions of q and kevin? Maybe give a full working example of your code to play with?

It might be possible to do some other optimizations.

I'm not too good on making parfor run, but try changing the inner loop to parfor and see if MATLAB complains.

[AA]

Kevin is a 1x60 matrix and q is a 100x 61 cell array. Each cell in q consists of matrices with million rows

[AA]

I changed the first line to parfor x = 1:100 and now the calculation is even slower. I thought parfor accelerates a calc on matlab but in my case it slows it down. Plz help.

[Sean de Wolski]

s is preallocated right?

[Matt J]

Kevin is a 1x60 matrix and q is a 100x 61 cell array.

The memory consumed by a 100x61 cell array, each containing a double precision matrix of size 1e6 x 60 is about 2.6 TB! The only way you could be storing such a matrix on any normal computer is if some of the q{x,y} are shared duplicates of each other. If that's true, you could speed things up by getting rid of the duplicates.

Answer 1

[Matt J]

There is no need to have a double loop. See if the following improves things. If not, tell us more about the contents of each q{x,y}, in particular how many columns it has.

           s=cell(size(q));

           parfor i=1:numel(q)

             b = abs(bsxfun(@minus,q{i},kevin))
             averagesAcrossColumns = mean(b, 2);
             b = abs(bsxfun(@minus,b,averagesAcrossColumns));
             b = sum(b, 2);
             s{i} = b;

           end

Answer 2

[Matt J]

If all q{x,y} have the same number of columns, you don't need a loop at all,

             b = abs(bsxfun(@minus,cat(3,q{:}),kevin))

              averagesAcrossColumns = mean(b, 2);
              b = abs(bsxfun(@minus,b,averagesAcrossColumns));
              b = sum(b, 2);

             s=reshape(num2cell(b,1),size(q));

[AA]

Error using cat Dimensions of matrices being concatenated are not consistent.

All columns of the matrices in the cell array q have 60 columns yet I still get the error.

[Matt J]

AA Commented:

It doesnt work. If I was to buy a graphics card supporting cuda how much faster would my parfor command become? Right now I need more than an hour to run this code because the cell array has matrices with millions of columns.

[Matt J]

Graphics cards have no effect on parfor. You could use gpuArray to speed things up, but first you need to explain the 2.6 TB RAM that your q array is consuming (see My Comment).

If you pad the rows of each q(x,y) so that they are all the same size, the code should work. The following test version (with fewer rows than you mention) runs for me in about 7 sec.

    q(1:100,1:61)={rand(1e3,60)};
    kevin=rand(1,60);

        tic;
                 b = abs(bsxfun(@minus,cat(3,q{:}),kevin));
                  averagesAcrossColumns = mean(b, 2);
                  b = abs(bsxfun(@minus,b,averagesAcrossColumns));
                  b = sum(b, 2);
                 s=reshape(num2cell(b,1),size(q));
        toc

[AA]

No, I saved the file and it is only around 5 gigabytes big. The columns of the matrices in the cell array are all the same but the rows of the matrices change even though the rows of the matrices across the same row in the cell array are the same.

[Matt J]

You said in your comment here "Each cell in q consists of matrices with million rows". If that's true, then q must contain a minimum of 1e6*60*100*61 values. In doubles, that is 2.6TB. So, you should correct that comment so we understand what's really going on.

Anyway, you haven't commented on my other Answer (with parfor). You also haven't commented on the possibility of padding all q(x,y) to have the same number of rows.

A GPU solution is starting to sound doubtful. You will need lots of RAM (<5GB) on the card and it will take a long time to transmit the data to the card.

[AA]

I am sorry for not being able to clarify myself. The original cell array had matrices with a few million rows. Then I reshaped the cell array and now it looks like this (see image)

</matlabcentral/answers/uploaded_files/22327/Untitled.png> The cell array is 61 columns wide and has 100 rows.