using ff2n(n) (two-level full factorial design) with n larger than 25 in my pc results in memory error

I want to get (two-level full factorial design) or all possible binary combination of number of bits larger than 25 but I get an error (the resulted matrix requires very large size). I want for for example to get the ff2n(30) in more than one matrix to solve this problem. I make my processing in the first matrix then remove that matrix and start processing on next one. I want to use ff2n because it's a very fast built-in function

[Benjamin Thompson]

What is fft2n? I cannot find information on this function in the MATLAB documentation.

[Mimo T]

Benjamin Thompson it's ff2n not fft2n

[dpb]

>> 2^30                             % number rows in design matrix
ans =
     1.073741824E+09
>>     
>> 2^30*30*8/1024/1024/1024         % memory required to hold one copy, GB
ans =
      240
>>

Even if you were to generate only a tiny fraction of the total at a time, it will take some billion iterations and if each consumes only 100 ms of compute time,

>> 2^30*100E-3/60/60/24/365
ans =
   3.404812988330796
>>

it would take 3-1/2 years to complete the task.

[Mimo T]

@dpb

I am aware of this issue and my computation time is much less than this.

Answer 1

[dpb]

Well, if you really can beat my estimate of the total time per iteration of generating the factors and then doing whatever it is you think you're going to be doing with them to get the final result by several orders of magnitude, then you could theoretically solve the problem, yes. I figured by the time you added in the logic to handle a piece-wise solution and whatever the other calculations are needed using that design matrix, that probably wouldn't be too bad of an estimate.

But, the basic idea isn't too hard; in fact, ff2n is just an m-file implemented by a for...end loop over each column in turn. It is built on the idea that one will be able to hold the whole design matrix in memory, granted, but the logic isn't difficult to build pieces of whatever size you wish -- look at a smaller design matrix to get the drift...

>> ff2n(4)
ans =
     0     0     0     0
     0     0     0     1
     0     0     1     0
     0     0     1     1
     0     1     0     0
     0     1     0     1
     0     1     1     0
     0     1     1     1
     1     0     0     0
     1     0     0     1
     1     0     1     0
     1     0     1     1
     1     1     0     0
     1     1     0     1
     1     1     1     0
     1     1     1     1
>> 

As expected, there are sixteen rows (2^4) by the four factors columns.

NB: the first column is simply [0 1] for half of the rows -- 0 for first half, 1 for second. Then, the second column is the same except for each half is split into half. The third is then split yet again until the last column simply alternates levels.

So, you pick whatever size you want to generate at a time and build the associated vector pattern needed for that subsection of each column.

[dpb]

Another trick you could use if you write your consuming functions to handle them would be to use int8 for the storage of the design matrix. Since is two-level, only need 1 bit to set/clear, but MATLAB doesn't implement a bit array data type, unfortunately.

[Mimo T]

@dpb "NB: the first column is simply [0 1] for half of the rows -- 0 for first half, 1 for second. Then, the second column is the same except for each half is split into half. The third is then split yet again until the last column simply alternates levels.

So, you pick whatever size you want to generate at a time and build the associated vector pattern needed for that subsection of each column."

That's what I am trying to make, I tried to edit .m code. I want, for your example, to generate only the first 4 rows, then the second 4 rows untill the last.

[dpb]

So, let's see what you tried and where it went bad...