Read and convert a .dat file containing binary data

Hi all,

I'm a beginner with the binary files and I would need some help. I have a *.dat file coming out from my CAEN digitizer that contains binary data. The structure of the file is the following:

EVENT=│Event│Event│Event│....│Event│; Event=(unsigned 64 bits int)Time Tag│(unsigned 16 bit)Energy

So the idea is: a 1-column (or 1-row, I'm not pretty sure) file containing data of timing and energy of alpha particles.

   - First Event: timing of the first particle in unsigned 64 bits int.
   - Second Event: Energy of the first alpha particle in unsigned 16 bit. 

   - Third Event: timing of the second particle in unsigned 64 bits int.
   - Fourth Event: Energy of the second alpha particle in unsigned 16 bit.

And so on....I would like to convert the file into a 2-columns file containing:

   - First column: timings
   - Second column: energies

First in binary to really see the zeros and ones and then to ASCII. I attach one of those files (I had to change the extension to .txt to upload).

Any help will be very welcome. Jose Manuel.

[dpb]

Didn't get the file attached...

Answer 1

[dpb]

EVENT=│Event│Event│Event│....│Event│;
Event=(unsigned 64 bits int)Time Tag│(unsigned 16 bit)Energy

In Matlab to read a file structure of differing sizes, you either have to loop over the file reading the proper size at each pass or use the memmapfile object to create the field structure.

A)

fid=fopen('yourfile');
te=[];
while ~feof(fid)
  te=[te; [fread(fid,1,'uint64') fread(fid,1,'uint16')]];
end
fid=fclose(fid);

Now you've got a Nx2 double array. To see the ASCII rep for first 10 values--

>> num2str(te(1:10,:),'%d')
ans =
 6892248    9310
17040576    9311
27189064    9309
37337248    9309
47485440    9318
57633536    9313
67781768    9318
77930056    9311
88078688    9313
98227224    9312
>>

or the hex or binary

>> [dec2bin(te(1:10,1)) ones(10,1)*' ' dec2bin(te(1:10,2))]
ans =
000011010010010101011011000 10010001011110
001000001000000010011000000 10010001011111
001100111101101111101001000 10010001011101
010001110011011100010100000 10010001011101
010110101001001001000000000 10010001100110
011011011110110101100000000 10010001100001
100000010100100010010001000 10010001100110
100101001010001111001001000 10010001011111
101001111111111100101100000 10010001100001
101110110101101010000011000 10010001100000
>>

B)

m = memmapfile('records.dat',        ...
       'Format', {                    ...
       'uint64' [1 1] 't';     ...
       'uint16' [1 1] 'e'});

The data is then accessed from the structure array m with the fields as above defined by m.data.t and m.data.e, respectively.

See the doc for both for more of the gory details...

[Jose Manuel Gomez Guzman]

Dear dpb,

thanks for the help. I have tried both methods and non of them work with my file. I attach it again. What I finally would like to have is a 2-column matrix like this example: 1359412216 (Timing)│7220 (Energy):

 -                              unsigned 64 bits int                         unsigned 16 bit 
 - 00110010 01110011 00110100 00111001 00110010 01101001 01111101 11111011   0001110000110100

I could not download your doc. Thanks in advance for your help, Jose Manuel.

[dpb]

By "see doc" I meant the installed help files on your machine--type

doc memmapfile

at the command line, for instance.

Anyway, sorry, I got too carried away with the attempt to concatenate two variables dynamically...the fread solution is correctly written as modified in the Answer section.

[Jose Manuel Gomez Guzman]

Dear dpb,

now I got it. I tried again the while loop and now it works perfectly fine. Thanks. The problem comes in the conversion to binary. For example, the last value of the first column (98227224) would be 00111001 00111000 00110010 00110010 00110111 00110010 00110010 00110100 (64 bits), meanwhile with dec2bin is 28 bits long. If I tried dec2bin(te(1:10,1),64) I got the same (101110110101101010000011000) but with (64-28) zeros. The same for the values in the second column: 9312 should be 00111001 00110011 00110001 00110010 (16 bits) and I have 10010001100000 (14 bits). With dec2bin(te(1:10,1),16) I get 0010010001100000.

Thanks for your help.

[dpb]

...(98227224) would be 00111001 00111000 00110010 00110010 00110111 00110010 00110010 00110100 (64 bits)...

How do you get that?

>> d=98227224
>> dec2hex(d)
ans =
[000000000]5DAD418

where I manually inserted the leading zero bytes/nybble (in brackets).

If you look at the raw data file via a dump utility, the 10th value is at offset 9*(8+2) = 90 --> 5A. Looking at the data from that point we see

            00 01 02 03 04 05 06 07  08 09 0A 0B 0C 0C 0E 0F
  ...
  0000 0050 .. .. .. .. .. .. .. ..  .. .. 18 D4 DA 05 00 00
  0000 0060 00 00 60 24 .. .. .. ..  .. .. .. .. .. .. .. ..

which are the 7 bytes of non-zero values plus the remaining nine zero bytes. That's all the significant bits there are in that value...it doesn't take close to 64 bits to store a value of roughly 10E8.

The above is a representation of the content of your file directly. Note there are only two bytes of data for the 16-bit value following.

[Jose Manuel Gomez Guzman]

Dear dpb,

my fault. I used one of those online converters but I made a mistake by assuming that 98227224 was ASCII and not decimal. Now it has sense. Thanks. Can you suggest any freeware dump utility to open the files?.

[dpb]

OK, I knew something had to be misunderstanding; couldn't figure out just what... :)

I don't have any freeware suggestions for a dump utility; I'm sure there must be any number around. I use the command-line shell products from JPSoft which includes a very nice one as a feature; consequently am not up on what else is around. On Windows unless it's been removed in versions since XP (yes, I'm still in the dark ages here :) ), there is DEBUG that can be used as a rudimentary dump utility, but it's not much of a tool for the purpose.

http://jpsoft.com/