Matlab data extraction speed up

Good day

I'm busy doing data extraction of a binary log file. The file is logged with various telemetry packets (with various classes with respective IDs).

I realised that reading a few bytes of the file at a time would be slow. So, what I did was read the entire file into a byte array. And then parse that byte array for the packets. This worked...

The problem is it is still slow.

I am pre-allocating memory and everything is stored in a multi-level struct called 'data'. An example data struct would be:

data.sensor.gps = some value. data.actuator.throttle.feedback = some other value. etc.

How can I speed this up? It takes about 13 minutes to extract a 40MByte file on my Intel Core 2 Duo with 4 Gigs of Ram.

I really need a solution in Matlab because my manager has requested I try doing in C as it will be faster.

Please help!!!

Amir

[Oleg Komarov]

Without code is hard to guess and help. Also, try to profile the code to see where the bottleneck is.

Answer 1

[Friedrich]

Hi,

could you give us an example how the data is stored in the binary file? And how should the data look like in the end?

This question is a little bit too fuzzy to give a good answer.

Please give us more detailed information so we can safe you from using C ;)

[Amir]

Thanks for the reply, Friedrich.

Well the data is the logged telemetry from our hardware. It can consist of various packets for different parameters. An example generic logged packet will be:

Start Chars (2byte) + Class (1 byte) + ID (1 byte) + length (2 bytes) + data ( n bytes) + Checksum (2 bytes).

Then, based on the data it is packed into a relevant field of the main data struct. So if the class is 0x04 (Sensors) and ID is 0x01 (GPS). Then the packet is packed into the struct like:

data.sens.gps.lat = data bytes (0 - 3) %% Latitude

data.sens.gps.long = data bytes (4 - 7) %% Longitude

data.sens.gps.alt = data bytes (8 - 11) %% Altitude

I read somewhere that structs are slow. Could this be the cause of my problem?

[Oleg Komarov]

Could be, but better profile the code for some files.

You can try to store it as:

data(1).sens

data(1).gps

data(1).lat

...

data(2).sens

data(2).gps

Where sens and gps constitute your combined ID.