Huge number of iterations
조회 수: 11 (최근 30일)
이전 댓글 표시
Hi Community members,
I am generating chemical formulas of compounds by forming combinations of elements and storing them in text file. The total number of combinations according to my calculations come out to be 18,217,382,400 i.e. i need 18,217,382,400 number of for loop iterations. I want to do this as quicky as possible. Please suggest an efficient method for doing this. I have tried both for and parfor, they take too long. A snippet of my code is shown below. I am using 2 workers and the code has been running for more than 24 hours now. How can I improve speed?
fcn = @() fopen( sprintf( 'chem_%d.txt', labindex ), 'wt' );
w = WorkerObjWrapper( fcn, {}, @fclose );
iterations=[length(a) length(b)]; % a and b are cell arrays. Length of a is 10944 length of b is 1664600
tic
parfor ix=1:prod(iterations)
ix
[d,e]=ind2sub(iterations,ix);
fprintf(w.Value, '%s\n', strcat(a{d},b{e}));
end
toc
clear w;
댓글 수: 6
Rik
2021년 9월 21일
Then you should probably consider buying computation time on some sort of cluster. If you don't tell us what you want to do, we can't suggest a way to avoid some of the computational work. Things take time. Sometimes the most efficient way is to reduce the number of things.
답변 (1개)
Walter Roberson
2021년 9월 20일
fcn = @() fopen( sprintf( 'chem_%d.txt', labindex ), 'wt' );
w = WorkerObjWrapper( fcn, {}, @fclose );
% a and b are cell arrays. Length of a is 10944 length of b is 1664600
b = b(:);
tic
iterations = length(a);
parfor ix=1:iterations
outs = strjoin(strcat(a(ix), b, {newline})); %a(ix) is deliberate in case a{ix} has whitespace
fwrite(w.Value, outs);
end
toc
댓글 수: 4
Walter Roberson
2021년 9월 22일
Huh. I really expected the fprintf version would be slower !
Notice that I build the fprintf format dynamically to include the current content from a . I assumed here that a does not contain any % characters.
NA = 100;
NB = 10000;
letters = ['A':'Z', '0':'9']; nlet = length(letters);
maxword = 5;
a = arrayfun(@(L) letters(randi(nlet, 1, L)), randi([1, maxword], 1, NA), 'uniform', 0);
b = arrayfun(@(L) letters(randi(nlet, 1, L)), randi([1, maxword], 1, NB), 'uniform', 0);
tn = tempname();
cleanME = onCleanup(@() delete(tn));
t1 = timeit(@() use_fprintf(tn, a, b), 0);
t2 = timeit(@() use_strjoin(tn, a, b), 0);
t3 = timeit(@() use_horzcat(tn, a, b), 0);
struct('fprintf', t1, 'strjoin', t2, 'horzcat', t3)
function use_fprintf(tn, a, b)
fid = fopen(tn, 'w');
for K = 1 : length(a)
fmt = sprintf('%s%%s\\n', a{K});
fprintf(fid, fmt, b{:});
end
fclose(fid);
dinfo = dir(tn);
fprintf('use_fprintf bytes = %d\n', dinfo.bytes);
end
function use_strjoin(tn, a, b)
fid = fopen(tn, 'w');
for K = 1 : length(a)
outs = strjoin(strcat(a(K), b, {newline}), '');
fwrite(fid, outs);
end
fclose(fid);
dinfo = dir(tn);
fprintf('use_strjoin bytes = %d\n', dinfo.bytes);
end
function use_horzcat(tn, a, b)
fid = fopen(tn, 'w');
for K = 1 : length(a)
temp = strcat(a(K), b, {newline});
outs = [temp{:}];
fwrite(fid, outs);
end
fclose(fid);
dinfo = dir(tn);
fprintf('use_horzcat bytes = %d\n', dinfo.bytes);
end
참고 항목
카테고리
Help Center 및 File Exchange에서 Loops and Conditional Statements에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!