speed up for loop
조회 수: 7 (최근 30일)
표시 이전 댓글
Hi everone. I have a problem with my code which was written in the following form. it take a long time and it is too time consuming.
In this code, I need to read 700 .hgt files with the size of 3600*3600 and make distincts 3*3 matrix from each of them.
I woulde be thankful if anyone could help me. Thanks in advance for your attenuation and help.
clc
clear all
close all
format long g
fid=fopen('list2.txt','r');
if fid==-1
disp('there is an error')
else
end
S = textscan(fid,'%s');
fclose(fid);
i=1;
fnames= S{i,i};
tic
NA=[];
for fiile=1:700
varargout = readhgt(fnames{fiile});
LAT=varargout.lat;LAT(end)=[];
LON=varargout.lon;LON(end)=[];
Z=varargout.z;Z(:,end)=[];
Z(end,:)=[];
% Z_vec=reshape(Z,[12967201,1]);
ROUGH=[];
for row=1:3:3597
for col=1:3:3597
A=[Z(row,col) Z(row,col+1) Z(row,col+2);Z(row+1,col) Z(row+1,col+1) Z(row+1,col+2);Z(row+2,col) Z(row+2,col+1) Z(row+2,col+2)];
Dif=(double(A)-mean(mean(A))).^2;
rough=sqrt(mean(mean(Dif)));
ROUGH=[ROUGH;mean([LAT(row) LAT(row+1) LAT(row+2)]) mean([LON(col) LON(col+1) LON(col+2)]) rough];
end
end
NA=[NA;ROUGH];
end
toc
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
actually the following part takes the most of time. in this part I am exploiting dependent 3*3 matrixs from a file with 3600*3600 elements.
for row=1:3:3597
for col=1:3:3597
A=[Z(row,col) Z(row,col+1) Z(row,col+2);Z(row+1,col) Z(row+1,col+1) Z(row+1,col+2);Z(row+2,col) Z(row+2,col+1) Z(row+2,col+2)];
Dif=(double(A)-mean(mean(A))).^2;
rough=sqrt(mean(mean(Dif)));
ROUGH=[ROUGH;mean([LAT(row) LAT(row+1) LAT(row+2)]) mean([LON(col) LON(col+1) LON(col+2)]) rough];
end
end
댓글 수: 0
채택된 답변
埃博拉酱
2023년 4월 7일
편집: 埃博拉酱
님. 2023년 4월 7일
When it comes to file I/O operations, parfor is usually not very helpful because the disk is always single-threaded. In contrast, eliminating for loops with vectorized methods can effectively improve performance.
fid=fopen('list2.txt');
if fid==-1
error('there is an error');
end
S = textscan(fid,'%s');
fclose(fid);
fnames= S{1};
tic
NA=cell(700,1);
for file=1:700
varargout = readhgt(fnames{file});
Z=varargout.z(1:3600,1:3600);
Z=std(reshape(Z',3,size(Z,1)/3,3,size(Z,2)/3),1,[1,3]);
[LAT,LON]=meshgrid(mean(reshape(varargout.lat(1:3600),3,[]),1),mean(reshape(varargout.lon(1:3600),3,[]),1));
NA{file}=[LAT(:),LON(:),Z(:)];
end
NA=vertcat(NA{:});
toc
Above I assume your files all contain 3601×3601 Z data. You can do more corrections and optimizations with your real data.
댓글 수: 6
埃博拉酱
2023년 4월 9일
@Walter Roberson For multiple files uniformly distributed across multiple drives, parallel I/O can theoretically improve performance, but this may require elaborated load balancing algorithms for these drives, which is very tricky. Judging from the code shown by OP, this is probably far beyond his programming ability.
추가 답변 (1개)
chicken vector
2023년 4월 7일
Try parallel computing:
% Get CPU's numebr of cores:
[~, numberOfCores] = evalc('feature(''numcores'')');
% Start parallel pool:
parpool(numberOfCores)
% Parallelise <for> loop:
parfor file = 1 : 700
% dostuff
end
Just be carefull and read the documentation about how to save restults from parfor iteratively, as you need to pre-allocate the output variable and you will have some other constraints.
참고 항목
카테고리
Help Center 및 File Exchange에서 Loops and Conditional Statements에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!