Trying to import multiple text files with odd formatting

So I have a years worth of data in the format of the file give and I only need 2 variables but I need to feed that data into Matlab, aside from manually doing so, is there a way to do so? Just reading the .txt file in to Matlab yields an unusable format. Thank you!

[Jan]

Please pot the code you use for "just reading" and share with the readers, what "unusable format" means. Which of the data are needed in which format? What have you tried so far?

[Cameron Power]

By "just reading" I mean importing the text file, and the unusable format is 3 columns and 20 rows, which does nothing to help with trying to isolate the WPSD and WDIR variables on the far right of the text file that I need. Most of the this is irrelevant however as the real question here is how to I edit ever text file on my mac so that I can isolate my needed variables.

[Jan]

@Cameron Power: You do not have to modify the files. As soon as you define clearly and uniquely, what you want to extract, it is possible to solve with some lines of Matlab code. I do not start to post a solution as long, as I have to guess, what you exactly need.

Do this manually for the posted example input.

[Cameron Power]

This is the relevent data that I need from each of the 6000 files.

note: Just the date and time and not the file start/end prefix is needed

Answer 1

[Image Analyst]

This code will work. It's a custom reader I wrote for you. It works at least for the one file you attached, and maybe for others depending on how much, if any, their format varies from the one you attached.

% Type file to command window.
fullFileName = 'test.txt';
type(fullFileName)  % OPTIONAL!

% Open the file.
fileID = fopen(fullFileName, 'rt');
% Read the first line of the file.
textLine = fgetl(fileID);
while ischar(textLine)
  fprintf('Processing line: "%s"...\n', textLine);
  if contains(textLine, 'File start time')
    fileStartTimes = sscanf(textLine, 'File start time : %f %f %f %f %f');
  elseif contains(textLine, 'File ending time')
    fileEndingTimes = sscanf(textLine, 'File ending time : %f %f %f %f %f');
  elseif contains(textLine, 'deg')
    % Read the next line which has the actual numbers we need on it.
    textLine = fgetl(fileID);
    numbers = sscanf(textLine, '%f %f');
    % Extract out WDIR and WSPD.
    WDIR = numbers(1);
    WSPD = numbers(2);
    % We're all done with this file, so break out (quit reading lines from it).
    break;
  end
  % Read the remaining lines of the file.
    textLine = fgetl(fileID);
end
% All done reading all lines, so close the file.
fclose(fileID);

[Cameron Power]

That worked brilliantly, thank you. I was hoping to store all the files into a cell array and then append new data in with each file, is there a way to do that? Thank you again.

-Also, I just tried it with another file and I received the error

Error using fgets Invalid file identifier. Use fopen to generate a valid file identifier.

Error in fgetl (line 32) [tline,lt] = fgets(fid);

[Image Analyst]

You can put it into a cell array, where k is your loop index:

ca{k, 1} = fileStartTimes; 
ca{k, 2} = fileEndingTimes;
ca{k, 3} = WDIR;
ca{k, 4} = WSPD;

but I'd recommend a structure array:

sa(k).fileStartTimes = fileStartTimes;
sa(k).fileEndingTimes = fileEndingTimes;
sa(k).WDIR = WDIR;
sa(k).WSPD = WSPD;

You'd have to figure out why the file it fails on is different than the standard file that you attached already. If you can't, then attach the bad file and we'll try to figure it out for you.

[Cameron Power]

Whichever way you think is more efficient works, but every file is almost identical to the format attached to this comment.

[Image Analyst]

That file is vastly different than the one you first gave. They're not even remotely the same! I think I gave you the general idea for how to read a custom format file line-by-line and to extract the numbers so I think you just need to follow my example and adapt it to this totally different format. It's such a complicated format that it would take me a long time to write a custom reader for it, so I think you can now do that just as well as I can, now that you know what to do. Good luck.

Answer 2

[Image Analyst]

See the FAQ for two code snippets to read in multiple files: https://matlab.wikia.com/wiki/FAQ#How_can_I_process_a_sequence_of_files.3F

In the loop, put your file reading code. It can use csvread(), importdata(), readtable(), or whatever.

[Cameron Power]

I`m sorry but the issue is not the importing of the files, it is the format of the table when I try to import the files. All I want to do is isolate the needed variables across all 5000+ text files.