Help definitely needed! How do I load multiple txt files and change the varible names of each to include the date from file

Hey everyone,

My coding skills have clearly dies on me with this one. I have a few txt files in directory that I am using textscan to load (massive files so I want to reduce memory). I would like a loop that goes through the list, loads in data into the workspace and gives each dataset a variable name that includes the date from the file name.

My file names are like this one midas_wind_198001-198012.txt from 1979-2013.

Here is the code I have thus far for loading but I am stuck on taking it further to renaming the variables. I do not want to use EVAL for efficiency reasons and I would like each dataset to be its own variable

fid = fopen('midas_wind_197701-197712.txt', 'r'); C = textscan(fid,'%s %s %f %f %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %s %f', 'Delimiter',','); fclose(fid);

Thanks a lot in advance!!

Answer 1

[Mischa Kim]

Masao, assuming that the only the year changes in each filename, use

 for ii = 1973:2013
   f_name   = strcat('midas_wind_',num2str(ii),'01-',num2str(ii),'12.txt');
   fid      = fopen(f_name, 'r'); 
   scan_val = textscan(fid,'%s ...   % !command cut off for readability!
   v_name   = genvarname(strcat('C',num2str(ii)));
   eval([v_name ' = scan_val';]);
   fclose(fid);
 end

[mashtine]

Hi Mischa,

Thanks for the quick reply. That looks like it will do that trick for reading in the files! I was hoping however to load in the files making each its own variable but is this right of me to think this way?

[Mischa Kim]

I see. Check the updated code above. The content of file

 midas_wind_198001-198012.txt

is now saved in the corresponding variable with name

 C1980

and accordingly for all other years.

[Jos (10584)]

You do not have to use EVAL. There are much better alternatives, like structs and cells ...

[Mischa Kim]

If you read Massao's question carefully, he states that he would like to "...give[s] each dataset a variable name that includes the date...". I assume your proposed solution using the likes of structs and cells does not do that.

[mashtine]

Thanks again Mischa and Jos. Yes, thats the tricky thing with EVAL, it does allow me to get more freedom with variable naming etc but I am been told structs may be best. However, I am dealing with files that each have ~ 1.2 million rows and contain both string and numeric data. I assumed keeping each file separate would then be best as I have 1980-2013 but do you suggest using structs instead? Or would that complicate things further down the line when indexing and doing calculations with the data?

You guys have been very helpful so I look forward to your reply!

[Jos (10584)]

It all depends on what your doing. If you only need to process a single file at a time, why not store the values into a variable that always has the same name.

filename = 'xxx.txt' 
A = load(filename)  % keeping a fixed name ... 
Result = Myfunction(A) % ... makes further coding easier
AnotherResult = MyOtherFunction(A)
disp(['A result of processing file: ' filename])
disp(Result)

I see no point in switching variable names here, as you have to edit the code every time. In humble my view, variables should be seen as coding elements that have a fixed name , but variable contents .

[Jos (10584)]

In addition, if you have two related variables, it is convenient to give them the same name but arrange them in an array:

Year(1) = 1990
Year(2) = 1996
Year(3) = 2001

Imagine using this design

Year1990 = 1990
Year1996 = 1996
Year2001 = 2001

Now see what happens if you need to change the third year

Year(3) = 1997  % easy to understand
Year2001 = 1997 % awkward!!

Answer 2

[Jos (10584)]

You can use cell or structs.

MyFiles = {'midas_wind_197701-197712.txt','whatever.txt','another name.txt'}

for k=1:numel(MyFiles)  % loop over all the files
    fid = fopen(MyFiles{k}, 'r'); 
    tmp = textscan(fid,'%s %s %f %f %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %s %f', 'Delimiter',','); 

    % store
    MyData{k} = tmp ;         % option 1
    MyDataToo(k).data = tmp ; % option 2
    MyDataToo(k).filename = MyFiles{k} ; % for convenience you can store the 
                                         % filename in a separate field
    fclose(fid);
end

Now the contents of MyData{x} or MyDataToo(k).data corresponds the file with the name MyFiles{x}

[mashtine]

Hey Jos,

I have used your code to create a cell array with the data and struct array. Unfortunately, both take up a large space when being saved (over 10 GB) whereas with Mischa method, my .mat file is only about 500 MB. Any reason why saving the variables separately causes this?