How do I process big datasets ??

조회 수: 2(최근 30일)
Add
Add 2016년 4월 5일
댓글: Walter Roberson 2016년 4월 5일
I have data of on file per day , that goes for each day of the month and then each month of the year. I now process daily files separately for daily results. How can I do this for complete analysis of full one year file. My file is like this YYYYMMDD.loc ; this means YYYY- Years, MM- months, DD-day. What if I want to take data of particular month and analyse ??

채택된 답변

Ced
Ced 2016년 4월 5일
편집: Ced 2016년 4월 5일
Hi
Do you now have each day in a separate file, or everything in one file?
Personally, if you want to evaluate different sections of time, I would keep each day in a separate file. Then, you can always just read the files you need.
E.g. Let's say you want to read the months of June and July of years 2013-2015:
(I am assuming that you want to read the same days and months for each year here)
years = 2013:2015;
months = 6:7;
days = [ 1 1 ; 30 31 ]; % first and last day to read, one month per column
N_years = length(years);
N_months = length(months);
N_days = sum(diff(days)+1); % Note: 1 is added to EACH month
N_files = N_years*N_months*N_days;
filenames = cell(N_files,1);
current_index = 1; % makes indexing easier
for i = 1:N_years
for j = 1:N_months
next_index = current_index + days(2,j)-days(1,j);
filenames(current_index:next_index,1) = ...
arrayfun(@(x)sprintf('%i%02i%02i.loc',years(i),months(j),x), days(1,j):days(2,j), 'UniformOutput',0);
current_index = next_index+1;
end
end
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end
Cheers
  댓글 수: 2
Walter Roberson
Walter Roberson 2016년 4월 5일
As Ced says,
Then, you can load all the data you need
for i = 1:N_files
% load data, process, combine.... whatever you need
end

댓글을 달려면 로그인하십시오.

추가 답변(0개)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by