I want to store a big matrix (a 720GB variable )

I want to store a big matrix. It needs about 720GB memory.

The big matrix is used to calculate eigs fuction (to find eigenvector) as a parameter.

Is it possible to get big matrix (720GB memory or even more) by using distributed computer environment?

I have ten-computers which has 72GB memory, so I want to save the big matrix divided to 10-computers.

Then do eigs function -> result = eigs('big matrix')

So the big matrix could be recognized as one variable even if it is stored in divided computers.

Is this possible?

[Oleg Komarov]

That's a HUGE matrix! How did you get that?

Answer 1

[Sarah Wait Zaranek]

I can think of two ways to do this:

1. If you have a sparse matrix, then I would use the sparse data type in MATLAB and probably use a 64bit machine/OS. This will not hold the non-zero values, greatly lessening your memory footprint. d = eig(S) accepts a sparse symmetric matrix S. Otherwise, use eigs in place of eig.

documentation on sparse

2. Using distributed arrays (across multiple machines). This will do what you suggest in your question, it will create a matrix whose elements are distributed across the memory of multiple machines but can still be treated as one big matrix. To do this you need to have the MATLAB Distributed Computing Server product installed on your cluster. Here is a demo using backslash - but eig should follow the same idea since it is supported for distributed arrays.

demo link

[Walter Roberson]

Unless something changed in 2011a, it is not possible to use eigs() to calculate all the eigenvalues of a sparse matrix: it errors out and says to use eig() instead.

Also, eigs is *very* slow if you are calculating more than a handful of eigenvalues

[Sarah Wait Zaranek]

Yes --> d = eigs(A) returns a vector of A's six largest magnitude eigenvalues by default.

It depends on what he needs.