how large matrix can I make?

I want to make a matrix in size of (10^60)X(10^6).

I know enough it isn't easy because of memory.

(futhermore, As I know, MATLAB has a limited matrix size.. right?)

I think there are many cases such as my problem.

If you are in this situation, what is the most reasonable solution?

Help me,plz TT

Answer 1

[Walter Roberson]

Even if the matrix only required one byte per entry (e.g., uint8), a matrix that big would require a processor able to handle a 220 bit address space. The largest address space available with commercial processors is 48 bit. You do not have any hope of using a matrix that big.

Is it possible that you want something that is (10^6)X(10^6) instead of (10^60)X(10^6) ?? At one byte per entry that would require less than a terabyte of memory, a 40 bit address space. It is possible to get systems with that much memory... but it is likely to be quite expensive.

We cannot really discuss reasonable solutions without more knowledge of what kind of calculations you are trying to do. Sometimes it is quite reasonable to "block" the calculations (apply the calculations to subsets of the data, and then work on the cross-block terms.)

By the way, how long were you anticipating that a single run of your algorithm should take?

Have you considered using parallel processing?

Answer 2

[park minah]

Thank you so much, Roberson. :)

I understand your answer well. As your advise, I used parallel computing, but it also doesn't work.TT (of course, It would be better.) In addition, I don't care no matter how long my running time is.

May I quest another one?

To solve this problem, I want that (10^60)X(10^6) matrix divides into a lot of small matrices. Do you know a proper key?

[Jan]

I think you are not sure about accepting the runtime. If you are able to process 1 million of the values per second, you will still need 3.1 * 10^52 years processing time. Even dividing this matrix into "a lot of" subarrays will not help to reduce this noticably. The problem exceeds the computer power currently installed on the earth.

[Walter Roberson]

Jan is correct: if you do want your matrix to be (10^60)X(10^6) then it is too large to finish the calculation within your lifetime. If one assumes that Moore's Law will continue to apply, that computing power will continue to double in speed for the foreseeable future, then there has been a proof demonstrated that it would not be worth even _starting_ your calculation until the last few hours of the project run-time on current equipment -- do not bother to start until some 10^50 years from now.

The exception to this would be if your matrix is *extremely* sparse, with the actual number of populated elements only in the billion range or less: it would be possible to emulate such a architecture on current systems.