How can I import repeating blocks of mixed data as a table?

I have a .dat file with many blocks of repeating mixed data. I would like to import such data preferably as a table for storing the data as column-given variables. I have tried some alternatives as the readtable, textscan, fgetl without success.

As an example of how the dataset looks like, there are 3 blocks below. From the first row of each block, only the data and time are of interest (e.g., 011031104000 reads as 01-10-31 10:40:00 that is year-month-day hour-minute-second).

Any help will be very welcome. Regards.

SNC 011031104000 UTC AVE 10 SMP 10.000 AZI 0.0 SDQ 100

x = 3.38 y = 3.00 z = -0.21 T = 17.06

xsig = 0.842 ysig = 0.616 zsig = 0.386 Tsig = 0.082

xycov = -0.058 xzcov = 0.136 xTcov = -0.022

yzcov = -0.046 yTcov = 0.003 zTcov = -0.009

psig = 0.706 qsig = 0.763 rsig = 0.395

tp = 0.155 tq = 0.168 tr = 0.087

ustar = -0.296 Tstar = Cd =-0.00427

MOs = mf = 0.106 hf = -11

u = 3.00 v = 3.38 w = -0.21

vel = 4.52 dir = 221.6

SNC 011031104010 UTC AVE 10 SMP 10.000 AZI 0.0 SDQ 100

x = 3.08 y = 3.17 z = 0.03 T = 17.10

xsig = 0.754 ysig = 0.688 zsig = 0.482 Tsig = 0.097

xycov = 0.224 xzcov = -0.118 xTcov = 0.029

yzcov = -0.105 yTcov = 0.016 zTcov = -0.015

psig = 0.861 qsig = 0.547 rsig = 0.485

tp = 0.194 tq = 0.123 tr = 0.109

ustar = 0.403 Tstar = -0.038 Cd = 0.00828

MOs = 0.002 mf = -0.197 hf = -18

u = 3.17 v = 3.08 w = 0.03

vel = 4.42 dir = 225.7

SNC 011031104020 UTC AVE 10 SMP 10.000 AZI 0.0 SDQ 100

x = 3.43 y = 2.36 z = -0.13 T = 17.10

xsig = 0.650 ysig = 0.766 zsig = 0.508 Tsig = 0.072

xycov = 0.001 xzcov = -0.075 xTcov = -0.004

yzcov = -0.209 yTcov = 0.014 zTcov = -0.008

psig = 0.699 qsig = 0.729 rsig = 0.496

tp = 0.167 tq = 0.175 tr = 0.119

ustar = 0.416 Tstar = -0.020 Cd = 0.00998

MOs = 0.001 mf = -0.210 hf = -10

u = 2.36 v = 3.43 w = -0.13

vel = 4.16 dir = 214.5

[Luna]

can you share the .dat file? Do you have "=" in your .dat file aswell?

[Stephen23]

This field value is very unfortunate:

Tstar =   

Are there really blank fields in the file?

[Luna]

I noticed the blanks you are right, it will be very hard to use delimiters with space and "=".

Answer 1

[Stephen23]

This imports and parses your example file:

str = fileread('Data.txt');
str = regexprep(str,{'=        ','[\r\n]+SNC','[\r\n]+'},{'= NaN ','/SNC',' '});
fmt = ['%s%s%s%s%f%s%f%s%f%s%f',repmat('%s=%f',1,31)];
opt = {'EndOfLine','/', 'MultipleDelimsAsOne',true};
C = textscan(str,fmt,opt{:});
V = [C{5:2:end}]
K = [C{4:2:end}]

You can easily convert K and V into a table:

>> T = array2table(V,'VariableNames',K(1,:))
T = 
    AVE    SMP    AZI    SDQ     x       y        z        T      xsig     ysig     zsig     Tsig     xycov     xzcov     xTcov     yzcov     yTcov    zTcov     psig     qsig     rsig      tp       tq       tr      ustar     Tstar        Cd        MOs       mf      hf      u       v        w      vel      dir 
    ___    ___    ___    ___    ____    ____    _____    _____    _____    _____    _____    _____    ______    ______    ______    ______    _____    ______    _____    _____    _____    _____    _____    _____    ______    ______    ________    _____    ______    ___    ____    ____    _____    ____    _____
    10     10     0      100    3.38       3    -0.21    17.06    0.842    0.616    0.386    0.082    -0.058     0.136    -0.022    -0.046    0.003    -0.009    0.706    0.763    0.395    0.155    0.168    0.087    -0.296       NaN    -0.00427      NaN     0.106    -11       3    3.38    -0.21    4.52    221.6
    10     10     0      100    3.08    3.17     0.03     17.1    0.754    0.688    0.482    0.097     0.224    -0.118     0.029    -0.105    0.016    -0.015    0.861    0.547    0.485    0.194    0.123    0.109     0.403    -0.038     0.00828    0.002    -0.197    -18    3.17    3.08     0.03    4.42    225.7
    10     10     0      100    3.43    2.36    -0.13     17.1     0.65    0.766    0.508    0.072     0.001    -0.075    -0.004    -0.209    0.014    -0.008    0.699    0.729    0.496    0.167    0.175    0.119     0.416     -0.02     0.00998    0.001     -0.21    -10    2.36    3.43    -0.13    4.16    214.5

and also include the SNC values if required:

>> T = [cell2table(strcat(C{2:3}),'VariableNames',C{1}(1)),T]
T = 
           SNC           AVE    SMP    AZI    SDQ     x       y        z        T      xsig     ysig     zsig     Tsig     xycov     xzcov     xTcov     yzcov     yTcov    zTcov     psig     qsig     rsig      tp       tq       tr      ustar     Tstar        Cd        MOs       mf      hf      u       v        w      vel      dir 
    _________________    ___    ___    ___    ___    ____    ____    _____    _____    _____    _____    _____    _____    ______    ______    ______    ______    _____    ______    _____    _____    _____    _____    _____    _____    ______    ______    ________    _____    ______    ___    ____    ____    _____    ____    _____
    '011031104000UTC'    10     10     0      100    3.38       3    -0.21    17.06    0.842    0.616    0.386    0.082    -0.058     0.136    -0.022    -0.046    0.003    -0.009    0.706    0.763    0.395    0.155    0.168    0.087    -0.296       NaN    -0.00427      NaN     0.106    -11       3    3.38    -0.21    4.52    221.6
    '011031104010UTC'    10     10     0      100    3.08    3.17     0.03     17.1    0.754    0.688    0.482    0.097     0.224    -0.118     0.029    -0.105    0.016    -0.015    0.861    0.547    0.485    0.194    0.123    0.109     0.403    -0.038     0.00828    0.002    -0.197    -18    3.17    3.08     0.03    4.42    225.7
    '011031104020UTC'    10     10     0      100    3.43    2.36    -0.13     17.1     0.65    0.766    0.508    0.072     0.001    -0.075    -0.004    -0.209    0.014    -0.008    0.699    0.729    0.496    0.167    0.175    0.119     0.416     -0.02     0.00998    0.001     -0.21    -10    2.36    3.43    -0.13    4.16    214.5

[Marlon Brancher]

Dear Stephen Cobeldick!

Thank you so much for your great work. You nailed it. Your solution worked perfectly for me.

Thanks also to Luna who tried to help.

Kind regards, Marlon.

[Luna]

+1 :)