Error using pdist2mex Error in kmeans>distfun

Hi, To represent our data (3233477*256) with Bag of visual word (BOW) which use KMeans clustering to extract visual words when we choose K=5000 this problem show:

Error using pdist2mex
Requested 3233477x5000 (120.5GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a
long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

Error in kmeans>distfun (line 747)
        D = pdist2mex(X,C,'sqe',[],[],[]);

Error in kmeans/loopBody (line 445)
        D = distfun(X, C, distance, 0, rep, reps);

Error in internal.stats.parallel.smartForReduce (line 136)
            reduce = loopbody(iter, S);

Error in kmeans (line 335)
ClusterBest = internal.stats.parallel.smartForReduce(...

Error in BOWHistogram (line 12)
[idx,c,sumd,D2] = kmeans(double(Tab_Feature_Data),NumClust);

What can I do to fix the error? Please advise me

[Rik]

Can you split the array into smaller parts? Unless you get a 120GB contiguous block of memory, you can't use this method. I'm not familiar enough with what you want to do to suggest a real solution.

[Tripoli Settou]

How can i do that i must extract the visual word from all data...how can i split it and get the visual word of all data?

Answer 1

[Bernhard Suhm]

You could try converting your large input data into a tall array (maybe as simple as t = tall(double(Tab_Feature_Data)), and then pass that tall array to kmeans. Though watch there are limitations which options of kmeans are available with tall arrays, see https://www.mathworks.com/help/stats/tall-array-support-usage-notes-and-limitations.html

[Tripoli Settou]

Hi Sir #Bernhard Suhm thanks for your reply... but when I try it this error show:

Error using tall/Kmeans
Too many output arguments.
Erro in BOW (line12)
[idx,c,sumd,D2] = kmeans(double(Tab_Features),NumClust);

[Tripoli Settou]

Also, The error in the question above is disappeared when I augmented the number of total size of the swap file but when I try my code with K = 1000 it's work without problem but when I choose K = 5000 the laptop is blocked however my laptop has 64 G of RAM and CPU E5-2620v3@2.40GHz ...I don't know why my laptop become so heavy, slow and blocked??!!

[Bernhard Suhm]

Your laptop will block unless you leverage multiple cores, but that also requires the Parallel Computing Toolbox. For large datasets like yours, you will need more powerful hardware or be patient with execution. The doc page I pointed you does point out that the 'tall' version of kmeans has only the first 3 output variables.

[Tripoli Settou]

So u advise me to use the Parallel Computing Toolbox to leverage multiple cores? right? also, I need the 4th output variable (D2) in next work

[idx,c,sumd,D2] = kmeans(double(Tab_Features),NumClust);

and can I use the Parallel Computing Toolbox without tall version KMeans i mean KMeans with Parallel Computing Toolbox?? it's work??

Answer 2

[carpcarp carpcarp]

我也遇到过这个问题，你可以看看你是不是命名了一个kmeans的m文件，如果有，并且是自己添加的，先把它删除（或者先发到qq等可以保存代码的应用后删除），然后你再去跑你的程序就可以成功了。

[carpcarp carpcarp]

You can see whether you have named a .m file called kmeans, like this:

if so, please delete it. Then run your program again.

Or if you don't know where is kmeans.m, you can download an application called everything.

My English level is limited, so I can only describe it simply.（←from translator)

[Walter Roberson]

(User points out that there can be problems if you accidentally have your own kmeans.m instead of using MATLAB's)