Improve efficiency of array multiplication

I hope someone can help me with improving the efficiency of my code. I have an array of 65536x65536 elements stored in a matfile. Now I would like to multiplicate each column with 65536 different constants. I have solved this as followed:

for ind=1:65536
            W = matObj_read.W(:,ind);  
            SW(ind)=sum(matObj_read.W(:,ind));
            WS = W.*SPS(ind);  
            SWS(ind) = sum(WS);
end

In the end the sum (SWS) is what I need to use

This code works, however, it takes like forever and since I need to repeat this calculation to often the total calculations time is arround a few years.... I have tried to make te code faster by removing the for loop and read the full variable W, but then I recieved an error for array size limit. This was simply solved but returned in an out of memmory error. I hope some can help me with making the code more efficient.

Answer 1

[David Young]

The value of SW is not used in the code you've shown. If you don't need the value of SW, it will help to omit this line. If you do need SW later, then you can probably speed up the computation by replacing the line that computes it with

SW(ind) = sum(W);

which avoids repeating the operations that get W.

It will also help to preallocate the arrays. That is, before the loop, do

sz = 65536;
SW = zeros(1, sz);
SWS = zeros(1, sz);

[Ramona Bouwman]

sorry, I needed to remove the SW(ind)= sum(matObj_read.W(:,ind)); line. I indeed pre-allocated the arrays as you suggested.

Answer 2

[Mahdiyar]

Hi Bouwman

There are different options that you can consider:

1- For heavy computation, you can try to employ all the core of the Computer/Laptop to be used for Matlab. Have a look at commands "parpool" and "parfor".

2- you can use the cammand repmat to multiply the different constant at the same time. Of course, in the code that you wrote variable "SPS" is not defined. Please look at the following Code.

Matrix = randi(10, 65536, 65536)
Constant = randi(10, 1, 65536)

Repeated_Constant = repmat(Constant, 65536, 1)
Final_Matrix = Matrix .* Repeated_Constant

In this code, I just put random number for Matrix and Constant number that you can substitute with your own matrix.

I run the code for number 4 instead of 65536, I get the following results that clearly explain.

Matrix =

     9     7    10    10
    10     1    10     5
     2     3     2     9
    10     6    10     2

Constant =

     5    10     8    10

Repeated_Constant =

     5    10     8    10
     5    10     8    10
     5    10     8    10
     5    10     8    10

Final_Matrix =

    45    70    80   100
    50    10    80    50
    10    30    16    90
    50    60    80    20

Regrads

[Ramona Bouwman]

Thank you very much for your repsons. I am looking in to your first solution. I expect that the seocond solution won't work due to the sizes... I must say that the line:

W = matObj_read.W(:,ind);

is the most time consuming

[David Young]

In general, bsxfun is likely to be more efficient than repmat .

[Ramona Bouwman]

I have implemnted the "parfor" command. The result is time is similar... I expect that this is due to reading of the matfile

[Mahdiyar]

The size of matrix is not the problem if your computer /laptop supports enough RAM. I recommend you to follow the second solution and also use comand lines of "tic" and "toc" before and after each command in order to know which line is taking the time.

Regards.

[Ramona Bouwman]

Thanks for your comment, I don't understand your comment the the matrix size is not the problem? If I try to read the whole matrix at once I recive an out of memmory error. Do you know how to solve this? The tic toc command learns me that the line:

W = matObj_read.W(:,ind);

takes the most time. If I could read the W matrix in once this would indeed be very helpfull! Thanks again for your reply I really appriciate this.

[Mahdiyar]

Hi

As I mentioned before: "The size of matrix is not the problem if your computer /laptop supports enough RAM ." I just wanted to make you sure that your laptop/computer is enough powerful.

Please correct me if I am wrong. I think that you want to read your matrix from an external file like Excel.

If you have the matrix in the workspace window, it would be better to divided into different parts and save them as different matrix matrix with different names and then load each of them separately and then follow you code.

Regards,

[Ramona Bouwman]

Hi,

My computer has 16.0 GB of RAM memory, so reading a 32 GB file (65536x65536 doubles ) is not an option. For your information: I did not read the matrix from an external file but estimated the matrix W iteratively with matlab. Reason for this was the size of the matrix W and the fact that I need to re-use this matrix several times. If I understand correctly you suggest to devide the matrix into different parts so that the use of the read command could be limited? I will try to implement this. Thanks again!

[Mahdiyar]

I do not know what kind of code you want to implement. But if I were you, I try to delete the variables which are not needed anymore.

Moreover, if your code can be divided in different parts,

1- run this part of code separately. It means that you can run your code untill this part and save the matrix matObj_read.W like bellow

save('matObj_read.W ', 'matObj_read.W ')

2- then clear all the variables and run this part of code that you mention at this question and save the final matrix.

3- Again delete all the variable and just load the final matrix and run the rest of code.

Running the First Part
.
.
.
.
clearvars -except matObj_read.W 
Runing this part of code
.
.
.
.
clearvars -except Final_Matrix
Running the rest of code

Regards,

[Ramona Bouwman]

Thank you, this indeed improved the efficiency. Only 1/3 of the original time is needed for the calculations.