Split the combvec resulting matrix?
조회 수: 4 (최근 30일)
이전 댓글 표시
I am using combvec function to create all possible combinations of 28 different numbers. The resulting matrix should have 3,221,225,472 columns (combinations) and 28 rows. I cannto run the code because of the out of memory error. I was wondering if there is anyway I can split the columns into more than one file to tackle this issue.
Here is the code:
a1 = ...=a26 = [1 2];
a27 = a28 = [1 2 3 4];
Mat = combvec(a1,..,a28).
Your help is highly appreciated.
댓글 수: 2
Stephen23
2023년 3월 4일
편집: Stephen23
2023년 3월 4일
"I cannto run the code because of the out of memory error."
That is not a big surprise if you ask for more than seven hundred and twenty gigabytes of contiguous memory:
R = 3221225472;
C = 28;
N = R*C
N*64 / 8 % bytes
To perform any operations on it you would then need three our four times that in practice, so does your computer have some three or four terabytes of memory installed? Note this is memory, not drive storage (i.e. not your HDD or SDD).
"I was wondering if there is anyway I can split the columns into more than one file to tackle this issue."
How exactly would splitting data up into separate arrays help "to tackle this issue" ? The amount of memory your computer has installed is constant: why would your computer suddenly have more memory if you ask it to store lots of smaller arrays?
What does any "file" have to do with this? A file is essentially some data saved to some storage device. What your storage devices have saved on them is unrelated to your computer's memory (unless you use some kind of slow virtual memory). You seem to be confusing storage devices (e.g. HDD, SDD) and memory.
In any case, rather than attempting things that do not make much sense and are very unlikely to work, you would be much better off reading the MATLAB documentation on how to store and manipulate large data:
and also reconsidering your algorithm: the best big data processing is the one that avoids the need for big data. If you can tweak your algorith so that you do not need to store all of that data at once in memory, then that is likely a better approach.
Dyuman Joshi
2023년 3월 4일
For splitting an array into columns you would require the array first, which you do not have.
The function combvec returns a matrix as an output, and you can not change how a function returns an output.
답변 (1개)
Walter Roberson
2023년 3월 4일
a1 = [1 2];
a2 = a1; a3 = a1; a4 = a1;
a27 = [1 2 3 4];
a28 = a27;
Mat = combvec(a1, a2, a3, a4, a27, a28);
numel(a1)^4 * numel(a27)^2
So with you using 26 vectors each of length 2, and two vectors of length 4, the number of columns should be
count = numel(a1)^26 * numel(a27)^2
whereas you are expecting
expected = 3221225472
expected / count
You are expected three times as many columns as should be calculated.
If you were expecting numel(a1)^26 * numel(a27)^2 then there is an easy way to sequence through the values:
for count = uint32(0) : uint32(2^26 - 1)
bits = bitget(count, 1:26) + 1;
thisiterelements = a1(bits); %assuming a1 through a26 are the same
now combine thisiterelements with all 16 combinations of a27 and a28 which you can prebuild
now do something with these 16 combinations.
end
You can reduce the overhead further by pre-building more of the trailing combinations
댓글 수: 0
참고 항목
카테고리
Help Center 및 File Exchange에서 Multidimensional Arrays에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!