Hello, sorry I missed adding that changing the inner loop to parfor(as all the variables are independent) alleviates the problem to an extent, but still very time consuming due to the load/save operations
efficient way to work with ~2k variables of size ~400x400x3
조회 수: 1 (최근 30일)
이전 댓글 표시
Hello,
I am running an optimization algorithm that involves iterating over ~2k variables, each variable representing an image of size ~400x400x3. In my iterations, I need to load these variables, update them, and go back to the next round, something like this
for iter=1:100
for var_no=1:1225
% Load var1_i
load(['var1_', num2str(var_no), '.mat');
% Load var2_i
load(['var2_', num2str(var_no), '.mat');
% Update var1_i, var2_i
var1_i = update_var1(var1_i);
var2_i = update_var2(var2_i);
% Save them again
save(['var1_', num2str(var_no), '.mat', var1_i, '-v7.3');
save(['var2_', num2str(var_no), '.mat', var2_i, '-v7.3');
end
end
As it can be expected, my code is extremely time consuming, and the size and their no. being very large, I cannot keep them in RAM. Can somebody please help me out and suggest me an efficient way to carry this? It has become a huge bottleneck in my run time. Thanks so much!
채택된 답변
Guillaume
2018년 2월 1일
"I cannot keep them in RAM"
Why not? 400x400x3x2000 is under 8 GB of memory as double. Don't you have that much memory available on a computer capable of running the parallel toolbox? If not, I'd really consider upgrading the memory.
Furthermore, since your processing images it's very possible that your images started as 8-bit integers, in which case keeping them all in memory would only require about 960 MB.
댓글 수: 5
Matt J
2018년 2월 2일
What format did your images start as? If they were 8-bit rgb image (the most common format), then keeping 5000 images of size 700x700(x3 colour channels) as uint8 would only require 8GB of data.
This assumes, though, that the iterative update process can tolerate uint8 precision.
추가 답변 (1개)
Matt J
2018년 1월 30일
댓글 수: 3
Matt J
2018년 2월 1일
편집: Matt J
2018년 2월 1일
I don't see that limitation in the matfile documentation. Where did you read it? The following simple example worked fine for me,
a = 0;
save tst1 a
save tst2 a
save tst3 a
m{1}=matfile('tst1','Writable',true);
m{2}=matfile('tst2','Writable',true);
m{3}=matfile('tst3','Writable',true);
parfor i=1:3
m{i}.a=i;
end
load tst1; a
load tst2; a
load tst3; a
참고 항목
카테고리
Help Center 및 File Exchange에서 Loops and Conditional Statements에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!