speed up for loop

조회 수: 7 (최근 30일)
Mina Mino
Mina Mino . 2023년 4월 7일
댓글: 埃博拉酱 . 2023년 4월 9일
Hi everone. I have a problem with my code which was written in the following form. it take a long time and it is too time consuming.
In this code, I need to read 700 .hgt files with the size of 3600*3600 and make distincts 3*3 matrix from each of them.
I woulde be thankful if anyone could help me. Thanks in advance for your attenuation and help.
clc
clear all
close all
format long g
fid=fopen('list2.txt','r');
if fid==-1
disp('there is an error')
else
end
S = textscan(fid,'%s');
fclose(fid);
i=1;
fnames= S{i,i};
tic
NA=[];
for fiile=1:700
varargout = readhgt(fnames{fiile});
LAT=varargout.lat;LAT(end)=[];
LON=varargout.lon;LON(end)=[];
Z=varargout.z;Z(:,end)=[];
Z(end,:)=[];
% Z_vec=reshape(Z,[12967201,1]);
ROUGH=[];
for row=1:3:3597
for col=1:3:3597
A=[Z(row,col) Z(row,col+1) Z(row,col+2);Z(row+1,col) Z(row+1,col+1) Z(row+1,col+2);Z(row+2,col) Z(row+2,col+1) Z(row+2,col+2)];
Dif=(double(A)-mean(mean(A))).^2;
rough=sqrt(mean(mean(Dif)));
ROUGH=[ROUGH;mean([LAT(row) LAT(row+1) LAT(row+2)]) mean([LON(col) LON(col+1) LON(col+2)]) rough];
end
end
NA=[NA;ROUGH];
end
toc
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
actually the following part takes the most of time. in this part I am exploiting dependent 3*3 matrixs from a file with 3600*3600 elements.
for row=1:3:3597
for col=1:3:3597
A=[Z(row,col) Z(row,col+1) Z(row,col+2);Z(row+1,col) Z(row+1,col+1) Z(row+1,col+2);Z(row+2,col) Z(row+2,col+1) Z(row+2,col+2)];
Dif=(double(A)-mean(mean(A))).^2;
rough=sqrt(mean(mean(Dif)));
ROUGH=[ROUGH;mean([LAT(row) LAT(row+1) LAT(row+2)]) mean([LON(col) LON(col+1) LON(col+2)]) rough];
end
end

채택된 답변

埃博拉酱
埃博拉酱 2023년 4월 7일
편집: 埃博拉酱 님. 2023년 4월 7일
When it comes to file I/O operations, parfor is usually not very helpful because the disk is always single-threaded. In contrast, eliminating for loops with vectorized methods can effectively improve performance.
fid=fopen('list2.txt');
if fid==-1
error('there is an error');
end
S = textscan(fid,'%s');
fclose(fid);
fnames= S{1};
tic
NA=cell(700,1);
for file=1:700
varargout = readhgt(fnames{file});
Z=varargout.z(1:3600,1:3600);
Z=std(reshape(Z',3,size(Z,1)/3,3,size(Z,2)/3),1,[1,3]);
[LAT,LON]=meshgrid(mean(reshape(varargout.lat(1:3600),3,[]),1),mean(reshape(varargout.lon(1:3600),3,[]),1));
NA{file}=[LAT(:),LON(:),Z(:)];
end
NA=vertcat(NA{:});
toc
Above I assume your files all contain 3601×3601 Z data. You can do more corrections and optimizations with your real data.
  댓글 수: 6
埃博拉酱
埃博拉酱 2023년 4월 9일
@Walter Roberson For multiple files uniformly distributed across multiple drives, parallel I/O can theoretically improve performance, but this may require elaborated load balancing algorithms for these drives, which is very tricky. Judging from the code shown by OP, this is probably far beyond his programming ability.

댓글을 달려면 로그인하십시오.

추가 답변 (1개)

chicken vector
chicken vector 2023년 4월 7일
Try parallel computing:
% Get CPU's numebr of cores:
[~, numberOfCores] = evalc('feature(''numcores'')');
% Start parallel pool:
parpool(numberOfCores)
% Parallelise <for> loop:
parfor file = 1 : 700
% dostuff
end
Just be carefull and read the documentation about how to save restults from parfor iteratively, as you need to pre-allocate the output variable and you will have some other constraints.
I suggest you to read the documentation and see some examples.
  댓글 수: 3
Mina Mino
Mina Mino 2023년 4월 7일
thanks for you response and help.

댓글을 달려면 로그인하십시오.

카테고리

Help CenterFile Exchange에서 Loops and Conditional Statements에 대해 자세히 알아보기

제품


릴리스

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by