How do I process big datasets ??
조회 수: 3 (최근 30일)
이전 댓글 표시
I have data of on file per day , that goes for each day of the month and then each month of the year. I now process daily files separately for daily results. How can I do this for complete analysis of full one year file. My file is like this YYYYMMDD.loc ; this means YYYY- Years, MM- months, DD-day. What if I want to take data of particular month and analyse ??
댓글 수: 0
채택된 답변
Ced
2016년 4월 5일
편집: Ced
2016년 4월 5일
Hi
Do you now have each day in a separate file, or everything in one file?
Personally, if you want to evaluate different sections of time, I would keep each day in a separate file. Then, you can always just read the files you need.
E.g. Let's say you want to read the months of June and July of years 2013-2015:
(I am assuming that you want to read the same days and months for each year here)
years = 2013:2015;
months = 6:7;
days = [ 1 1 ; 30 31 ]; % first and last day to read, one month per column
N_years = length(years);
N_months = length(months);
N_days = sum(diff(days)+1); % Note: 1 is added to EACH month
N_files = N_years*N_months*N_days;
filenames = cell(N_files,1);
current_index = 1; % makes indexing easier
for i = 1:N_years
for j = 1:N_months
next_index = current_index + days(2,j)-days(1,j);
filenames(current_index:next_index,1) = ...
arrayfun(@(x)sprintf('%i%02i%02i.loc',years(i),months(j),x), days(1,j):days(2,j), 'UniformOutput',0);
current_index = next_index+1;
end
end
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end
Cheers
댓글 수: 2
Walter Roberson
2016년 4월 5일
As Ced says,
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end
추가 답변 (0개)
참고 항목
카테고리
Help Center 및 File Exchange에서 Data Type Identification에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!