import multiple .dat files

Hello,

I've been struggling with this problem for several days now. Tried different solutions and I think I'm almost there, except for following error:

PROBLEM: I have a folder with 500 .dat files, they each consist of 1 header line with 40 variables (strings) and about 8000 lines with recorded integer data. I want to import them into matlab so I have 40 vectors of (500*8000) by (1).

I use this code:

function importfiles()

datfiles=dir('*.dat'); 
numfiles=length(datfiles);

%Import the files

for k=1:length(datfiles);
  newData1 = importdata(datfiles(k).name);

    A=newData1.data; %Create matrix A, with all data from the file 
    if k==1
        B=zeros(size(A));
        B=vertcat(B,A);
    else
        B=vertcat(B,A);  %vertically concatenates matrix A to B
    end
end
     B(all(B==0,2),:)=[];   %Delete all the zero lines

% Create new variables in the base workspace from those fields.

for i=1:size(newData1.colheaders, 2)
    assignin('base', genvarname(newData1.colheaders{i}), B(:,i));
end

When I execute this I get following error:

Attempt to reference field of non-structure array.

Error in importfiles (line 15) A=newData1.data;

!!!!When I execute for_ k=1:60;_ it works.!!!!!

What is causing this problem for k=1:(integer bigger than 60)???

Answer 1

[per isakson]

Set

    dbstop if error

and inspect the values of the variables at line 15

[VatoG]

With your answer I found out that in file 64 there is one parameter that is ERROR for a few hundred rows. After checking, this problem seems to repeat for a few of my .dat files.

So for matlab this is a string instead of a double. How can I change my code such that matlab sees this as a NaN?

[Walter Roberson]

Use textscan() if your file is text-based. Use xlsread() and examine only the numeric matrix if your file is .csv or .xls or .xlsx .

[VatoG]

It's a .dat file, ASCII-code, with space as a delimiter. I tried textscan() but it didn't really worked out because the output was a bunch of vectors with strings in it. Since the main part (90%) of my data is doubles I think importdata() works better.

I can however change the .dat files. I can replace the ERROR-string by something, but in matlab I would like to see this as a NaN, what value as input gives a NaN as an output? I tried a space, but since my delimiter is a space, that doesn't work.

[per isakson]

Try

    str = fileread( file_spec );
    str = strrep( str, 'ERROR', 'nan' );
    fprintf( fid, '%s', str )

and read the new file. Or

    cac = textscan( fid, format, ..., 'TreatAsEmpty', {'ERROR'} )

NOT tested