MATLAB Answers

Trying to arrange big data

조회 수: 22(최근 30일)
hi everyone, im having trouble to reach max speed reading huge csv files, and i would like to hear your ideas
i'm using datastore in order to pre - arrange 1910 headers, and than i want to fix my output to hold them in a specific struct, and start reading individually because not all 1910 headers are full of data. my main problem is that every header seperated with '_' and it makes it hard to read.
for example:
FruitDs = datastore(Fruits.csv) %im keeping it short, but the function is going fine
NumOfHeaders = length(FruitDs.Variablenames); %e.g Headers : "Apples_colour_S1_red_dated" "Apples_colour_S1_red_dated" "Apples_colour_S_green_dated"....
for n = 1:NumOfHeaders
if strfind(Ds.VariableName{n},'Apples')
tmp = Ds.VariableName{n};
A =strfind(tmp,'_');
Ds.SelectedVariableName = Ds.VariableName(n);
ApplesData = readall(Ds);
eval([tmp '= ApplesData']); % the struct i need is that out.Apples.colour_XXXXX will contain all data of the specific apple
Fruits.Apples = tmp;
this function works fine, so my questions as follows:
  1. is there any faster way to do it?
  2. do you have a smart and fast logical way to avoid reading empty headers (because 1910x390000 can be too much and not all of them are full (i filled them with NA in the datastore..))
  3. i have some cases which the headers are different only by number, and i do want to seperate them. let's say "Apples_colour_S1_....", "Apples_colour_S2_...". is there a way to avoid second loop (loop that runs over all the SX)?
thanks in advance

채택된 답변

Walter Roberson
Walter Roberson 3 Mar 2021
don't use eval there. Use setfield. Split the name at _ into a cell array and do cell expansion to insert the field names
  댓글 수: 3
harel yadid
harel yadid 9 Mar 2021
thank you very much!

댓글을 달려면 로그인하십시오.

추가 답변(0개)




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by