out of memory: array too long

조회 수: 10(최근 30일)
Win co
Win co 2014년 9월 24일
댓글: Win co 2014년 9월 25일
Hello, I'd like to run function pdist of Matlab on an array whose the length is N*(N-1)/2 where N=340000. Matlab is out of memory to preallocate this array. Could anyone give me a solution please? Kind regards, Winn
  댓글 수: 1
Oleg Komarov
Oleg Komarov 2014년 9월 24일
As per the you reference in http://www.mathworks.co.uk/matlabcentral/answers/156028#comment_239068, you can block process and keep partial sums. In any case you will need to use for loops.

댓글을 달려면 로그인하십시오.

채택된 답변

Sean de Wolski
Sean de Wolski 2014년 9월 24일
That's only going to require 57.8 gigabytes of memory for a single column. And that's the end result. Surely there'll be some large intermediate arrays as well.
I'd recommend downsampling or chunking up the calls.
Do you need every pairwise distance? Are you looking for something specific? What's the end goal?
  댓글 수: 8
Win co
Win co 2014년 9월 25일
I'm doing my computation on cluster server, I'll delete these files once comparing all distances between files is done. I could give a simple example of how my code works if you want. Thanks sincerely for all your taking time to my problem.

댓글을 달려면 로그인하십시오.

추가 답변(1개)

Adam
Adam 2014년 9월 24일
Depending on how much over memory it is you could try converting your data to single before you pass it to pdist. That should take half the memory.
I don't know off-hand if pdist is overloaded for integer types or not. If it is then you could also use them depending what level of accuracy you requie.
  댓글 수: 2
Adam
Adam 2014년 9월 24일
Well, yes, but that creates an array of doubles. You can try pre-allocating:
dist = zeros(N*(N-1)/2,1, 'single')
or even
dist = zeros(N*(N-1)/2,1, 'uint8')
but the latter option assumes pdist works on uint8 data and that you really don't care much about accuracy!

댓글을 달려면 로그인하십시오.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by