Out of memory error for big data

조회 수: 3 (최근 30일)
Sharif Khalil
Sharif Khalil 2020년 12월 29일
편집: Sharif Khalil 2020년 12월 31일
The masterDataMatrxOffset, contains 464780 rows and 256 columns of complex numbers. I use meshgrid to get 256X256 matrix of the real and imaginary and store it in a cell, but I get this error: Is there a way to overcome the out of memory?
Out of memory.
Error in meshgrid (line 62)
yy = repmat(ycol,size(xrow));
Error in auto_encoder (line 34)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
More information
tic;
clear; clc; close all;
%% Load Files
load masterDataMatrxOffset;
%% Extract Parameters
iAntenna = size(masterData,2) - 16;
data = masterData(:,1:iAntenna);
power = normalize(real(data),'range');
phase = normalize(imag(data),'range');
Region = masterData(:,261);
iReceiver = length(data);
iRegion = max(Region);
%% Initializations
xTrn = cell(1,max(Region));
xTst = cell(1,max(Region));
datamat = cell(1,size(data,1));
%% Split into training and testing
for idx = 1:iRegion
IDX = find(Region==idx);
train = sort(randperm(size(IDX,1),ceil(0.8*size(IDX,1))));
trnidx = IDX(train);
tstidx = setdiff(IDX,trnidx);
xTrn{idx} = trnidx;
xTst{idx} = tstidx;
end
xTrnidx = cell2mat(xTrn');
xTstidx = cell2mat(xTst');
Trainreg = zeros(numel(Region), max(Region));
Trainreg(sub2ind(size(Trainreg), 1:numel(Region), Region')) = 1;
tTrain = Trainreg';
for obsrv = 1:size(power,1)
sprintf('observation = %d',obsrv);
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
datamat{obsrv} = Po.^2 + Ph.^2;
end
toc;
  댓글 수: 4
Ive J
Ive J 2020년 12월 29일
편집: Ive J 2020년 12월 29일
You still can store them in a cell array
myCell = {};
myCell{1} = tall(1:100); myCell{2} = tall(101:200);
myCell{3} = myCell{1}.^myCell{2};
myCell =
1×3 cell array
{1×100 tall} {1×100 tall} {1×100 tall}
Sharif Khalil
Sharif Khalil 2020년 12월 31일
편집: Sharif Khalil 2020년 12월 31일
It still shows out of memory. The cell is 1X464780, but the matrix inside each cell is 256X256.
for obsrv = 1:size(power,1)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
Talldata = tall(Po.^2 + Ph.^2);
datamat{obsrv} = Talldata;
end

댓글을 달려면 로그인하십시오.

답변 (0개)

카테고리

Help CenterFile Exchange에서 Large Files and Big Data에 대해 자세히 알아보기

태그

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by