Accelerometer data file correction

Hi,

My accelerometer writes a .csv log file continuously for 24 h a day that I read into Matlab with the importdata function. Those files contain 10 header lines beginning with a semicolon and the rest is numerical data organized into 4 columns (time,x-axis acceleration, y-axis, z-axis).

My new sensors write an error line sometimes into the data section which also start with a semicolon. Therefore, the importdata function stops reading the data at the point where the error message appears.

Could you please help me in solving this question? I want to read in the file, delete the error message lines and save the file with its original name. csvread and dlmread were not useful in solving this question.

Thanks!

Answer 1

[Walter Roberson]

There are multiple ways. One way is:

filestr = fileread('YourData.csv');

filestr = regexprep(filestr, '^;.*$\n', '', 'lineanchors', 'dotexceptnewline');

fid = fopen('YourNewData.csv','w');
fwrite(fid, filestr);
fclose(fid)

[Josee Seigneur]

Thank you very much! The solution is almost perfect now.

THe remaining issue is that the file must to contain 432.000 data rows.

Since we delete rows which contain an error message in the data section we also lose information that would normally belong to that data point. The result is a shorter file than 432.000 rows.

The data does not change too much from row to row. Therefore, the x,y,z data and the timestamp could be interpolated by taking the mean of the previous and the next row.

Can you build that into your great solution?

Thanks!

[Walter Roberson]

filestr = regexprep(filestr, '^;.*$', 'NaN,NaN,NaN,NaN', 'lineanchors', 'dotexceptnewline');

That would introduce placeholders for the missing data, allowing you to use any available function to interpolate data in place of NaN.

One point that I would question is whether an error message always replaces _exactly_ one point ? If several might be replaced, then linear interpolation would not be suitable as the number of missing points would not be known. The code I give here will replace each error message with one row of NaN, which will help if each there is one error message per missing point.

My first guess would be that error messages in the log would tend to be associated with extreme acceleration values -- times at which the continuity of the acceleration data would be least likely ?

[Josee Seigneur]

Thank you!