Read Data From ADTF DAT Files

This example shows how to read data from ADTF DAT files using adtfFileReader and adtfStreamReader objects.

To read the data, you first create an adtfFileReader object that will act as a file handler object for the DAT file. This object gives useful information about the streams present in the DAT file and its contents. Following are the three main ways you can create an adtfFileReader object:

Using DAT file only: Applies to image and video stream data
Using DAT file and DDL description file: Applies to structured data
Using ADTF plugins, given a DAT file and/or DDL description file: Applies to data which requires some ADTF plugins

Next, you select the data to be read from the DAT file, which creates an adtfStreamReader object. You can then use this object to read the data. Following are some of the common ways you can read the data:

The data you read from the DAT file will be present in the MATLAB workspace, in raw form.

Create `adtfFileReader` Object

Using DAT file only

Create an adtfFileReader object by specifying ADTF DAT file name as the only argument to create the file reader object, to read data from streams like images and videos.

datFileName = fullfile("C:","data","sample_can_video.dat");
fileReader  = adtfFileReader(datFileName) %#ok

fileReader = 
              DataFileName: "C:\data\sample_can_video.dat"
       DescriptionFileName: ""
           PluginDirectory: ""
               StreamCount: 2
                StreamInfo: 

    StreamIndex    StreamName      StreamType      StartTime     EndTime      ItemCount    SubstreamInfo
    ___________    __________    ______________    _________    __________    _________    _____________

         1         {'rawcan'}    {'UNRESOLVED'}      0 sec      14.805 sec       743       {0×1 struct} 
         2         {'video' }    {'adtf/image'}      0 sec      14.799 sec       149       {0×1 struct}

From the StreamInfo property, note that

Stream 1 is named 'rawcan'. It contains 743 data items spanning from 0 to 14.805 seconds. However, this stream is not supported for reading as its 'StreamType' is 'UNRESOLVED'. To read such streams, we might need to some additional ADTF plugins, as explained here.
Stream 2 is named ‘video’. It is an 'adtf/image' stream, which is a common ADTF stream type for video and image streams. It contains 149 image frames, recorded over an interval of 14.799 seconds.
Both streams do not contain any substreams, hence the 'SubstreamInfo' field has an empty struct.

Using DAT file and DDL description file

Some DAT files contain structured data (for example, CAN data). To read such data into MATLAB workspace, you need a DDL description file containing details about the structure of the data within the streams. Specify the name of the DDL description file as an additional argument while creating an adtfFileReader object. Note that the DAT file sample_can.adtfdat contains dummy data and is used for demonstration purposes only.

datFileName = fullfile("C:","data","sample_can.adtfdat");
ddlFileName = fullfile("C:","data","sample_can.description");
fileReader  = adtfFileReader(datFileName, ddlFileName) %#ok

fileReader = 
              DataFileName: "C:\data\sample_can.adtfdat"
       DescriptionFileName: "C:\data\sample_can.description"
           PluginDirectory: ""
               StreamCount: 1
                StreamInfo: 

    StreamIndex    StreamName         StreamType          StartTime    EndTime     ItemCount    SubstreamInfo
    ___________    __________    _____________________    _________    ________    _________    _____________

         1         {'rawcan'}    {'adtf/devicetb/can'}      0 sec      0.98 sec       99        {0×1 struct}

Note that the above output shows a different stream type, 'adtf/devicetb/can', for the CAN stream, unlike the 'UNRESOLVED' stream type in the previous section. This due to the fact that the sample_can.adtfdat is in ADTF 3.0 format file whereas sample_can_video.dat is an ADTF 2.0 format file. For the 2.0 format, additional plugins may be necessary to read the data.

In some cases, DDL can be stored internally into a DAT file itself. Then the DDL description file is not required.

Use ADTF Plugins, Given a DAT File and/or DDL Description File

An ADTF Plugin is a compiled object that provides additional functionality to ADTF Runtime. They are very specific to ADTF framework and you can read more about them here.

In certain cases, ADTF Plugins are necessary to read data from streams. For such cases, specify the path to the folder storing the plugins as an additional argument while creating an adtfFileReader object. Replace the value of pluginFolder variable with the path on your system that contain the plugins.

datFileName  = fullfile("C:","data","sample_can_video.dat"); 
ddlFileName  = fullfile("C:","data","sample_can_video.description");
pluginFolder = fullfile("C:",'pluginFolder');

fileReader  = adtfFileReader(datFileName, ddlFileName, pluginFolder) %#ok

fileReader = 
              DataFileName: "C:\data\sample_can_video.dat"
       DescriptionFileName: "C:\data\sample_can_video.description"
           PluginDirectory: "C:\pluginFolder"
               StreamCount: 2
                StreamInfo: 

    StreamIndex    StreamName         StreamType          StartTime     EndTime      ItemCount    SubstreamInfo
    ___________    __________    _____________________    _________    __________    _________    _____________

         1         {'rawcan'}    {'adtf/devicetb/can'}      0 sec      14.805 sec       743       {0×1 struct} 
         2         {'video' }    {'adtf/image'       }      0 sec      14.799 sec       149       {0×1 struct}

Note that there are different versions of same plugin for every Operating System. If plugins are required to read streams in the DAT file, and you do not specify their path, the StreamType value for those will be ‘UNRESOLVED’.

For cases where DDL description file is not required, but a plugin is needed, then you can use the following syntax:

datFileName  = fullfile("C:","data","sample_can_video.dat"); 
pluginFolder = fullfile("C:",'pluginFolder');

fileReader  = adtfFileReader(datFileName, pluginFolder) %#ok

fileReader = 
              DataFileName: "C:\data\sample_can_video.dat"
       DescriptionFileName: ""
           PluginDirectory: "C:\pluginFolder"
               StreamCount: 2
                StreamInfo: 

    StreamIndex    StreamName         StreamType          StartTime     EndTime      ItemCount    SubstreamInfo
    ___________    __________    _____________________    _________    __________    _________    _____________

         1         {'rawcan'}    {'adtf/devicetb/can'}      0 sec      14.805 sec       743       {0×1 struct} 
         2         {'video' }    {'adtf/image'       }      0 sec      14.799 sec       149       {0×1 struct}

Select and Read Data

Read Data from Single Stream

Create the adtfFileReader object. Note that the DAT file sample_struct.dat contains dummy data and is used for demonstration purposes only.

datFileName = fullfile("C:","data","sample_struct.dat");
ddlFileName = fullfile("C:","data","sample_struct.description");
fileReader  = adtfFileReader(datFileName, ddlFileName)

fileReader = 
              DataFileName: "C:\data\sample_struct.dat"
       DescriptionFileName: "C:\data\sample_struct.description"
           PluginDirectory: ""
               StreamCount: 2
                StreamInfo: 

    StreamIndex       StreamName          StreamType       StartTime    EndTime     ItemCount    SubstreamInfo
    ___________    ________________    ________________    _________    ________    _________    _____________

         1         {'FirstStream' }    {'adtf2/legacy'}    0.09 sec     1.07 sec       99        {0×1 struct} 
         2         {'SecondStream'}    {'adtf2/legacy'}    0.09 sec     0.98 sec       90        {0×1 struct}

Select the stream to be read by specifying their stream index.

streamReader = select(fileReader, 1);

Read the first item in the selected stream.

dataItem = readNext(streamReader)

dataItem = struct with fields:
    StreamIndex: 1
           Data: [1×1 struct]

In the structure dataItem, 'StreamIndex' field shows the selected stream index and the 'Data' field.

disp(dataItem.Data);

     ChunkTimestamp: 90000
    SampleTimestamp: 90000
            IsValid: 1
               Item: [1×1 struct]
           ItemName: 'tFirstStream'

Structure item.Data, contains the actual data 'Item', and 'ItemName' representing the struct name given to the data item when the DAT file was created. 'ChunkTimestamp' (in microseconds) is the time at which this data was written into the DAT file and 'SampleTimestamp' (in microseconds) is the time at which this data was recorded or computed (let us say from a sensor). 'IsValid' when set to logical(1) means that the data inside 'Item' is valid, otherwise at logical(0) it will contain a string of error message explaining why a valid data is not extracted.

% Display time at which data was written
fprintf("ChunkTimestamp = %d\n",dataItem.Data.ChunkTimestamp);

ChunkTimestamp = 90000

% Display time at which data was created
fprintf("SampleTimestamp = %d\n",dataItem.Data.SampleTimestamp);

SampleTimestamp = 90000

% Display data
disp(dataItem.Data.Item)

    signal1: [1×1 struct]
    signal2: [2×1 double]
    Signal1: [1×1 struct]
    Signal2: [2×1 double]

You can also iterate over all the data items in the selected stream.

% Read one item at a time
while hasNext(streamReader)
    dataItem = readNext(streamReader);
    % Process data
end

Alternatively, you can read all data items at once and iterate over it later.

% Read everythin at once
items = read(streamReader);

% Iterate over the data
for i=1:streamReader.DataCount
    timestamp = items.Data(i).ChunkTimestamp;
    data = items.Data(i).Item; 
    % Process data
end

Read Data with `TimeRange` and `IndexRange` Filters

Create the adtfFileReader object.

datFileName = fullfile("C:","data","sample_can.adtfdat");
ddlFileName = fullfile("C:","data","sample_can.description");
fileReader  = adtfFileReader(datFileName, ddlFileName)

fileReader = 
              DataFileName: "C:\data\sample_can.adtfdat"
       DescriptionFileName: "C:\data\sample_can.description"
           PluginDirectory: ""
               StreamCount: 1
                StreamInfo: 

    StreamIndex    StreamName         StreamType          StartTime    EndTime     ItemCount    SubstreamInfo
    ___________    __________    _____________________    _________    ________    _________    _____________

         1         {'rawcan'}    {'adtf/devicetb/can'}      0 sec      0.98 sec       99        {0×1 struct}

Use the name-value argument, IndexRange, in the select function to filter the search to the last 10 data items in the selected stream.

streamIndex  = 1;
startIndex   = fileReader.StreamInfo(streamIndex).ItemCount - 9; % 10th element index from last
endIndex     = fileReader.StreamInfo(streamIndex).ItemCount; % last index
streamReader = select(fileReader, streamIndex, IndexRange=[startIndex endIndex]); %#ok

Use the name-value argument, TimeRange, to filter the search to all the data items recorded between 1 to 2 seconds, across the selected streams.

startTime    = seconds(0.1);
endTime      = seconds(0.2);
streamReader = select(fileReader, TimeRange=[startTime endTime]); %#ok

INFO : All streams are selected.

Next, you can iterate through the items using the readNext and hasNext functions, or read all items at once using readNext function. See Read Data from Single Stream section for more information.

Reading multiple streams

Create the adtfFileReader object.

datFileName = fullfile("C:","data","sample_struct.dat");
ddlFileName = fullfile("C:","data","sample_struct.description");
fileReader  = adtfFileReader(datFileName, ddlFileName)

fileReader = 
              DataFileName: "C:\data\sample_struct.dat"
       DescriptionFileName: "C:\data\sample_struct.description"
           PluginDirectory: ""
               StreamCount: 2
                StreamInfo: 

    StreamIndex       StreamName          StreamType       StartTime    EndTime     ItemCount    SubstreamInfo
    ___________    ________________    ________________    _________    ________    _________    _____________

         1         {'FirstStream' }    {'adtf2/legacy'}    0.09 sec     1.07 sec       99        {0×1 struct} 
         2         {'SecondStream'}    {'adtf2/legacy'}    0.09 sec     0.98 sec       90        {0×1 struct}

You can select streams by specifying their stream indices. To select all streams by default, then do not specify any stream indices.

While reading data from multiple streams simultaneously, it is possible that there are unequal number of data items across different streams. To illustrate, perform the following selection.

firstStreamIndex  = 1; 
secondStreamIndex = 2;
startTime = seconds(0.98);
endTime   = seconds(2.0);

streamReader = select(fileReader, [firstStreamIndex, secondStreamIndex], TimeRange=[startTime endTime]);

fprintf("Number of elements in stream 1 = %d\n",streamReader.DataCount(firstStreamIndex));

Number of elements in stream 1 = 10

fprintf("Number of elements in stream 2 = %d\n",streamReader.DataCount(secondStreamIndex));

Number of elements in stream 2 = 1

Note that first stream has 10 items and second stream has only 1 item. If you read all data items at once using the read function, then stream 1 will return an array of 10 structures, and stream 2 will return a single structure.

allData = read(streamReader)

allData=2×1 struct array with fields:
    StreamIndex
    Data

When you read data one-by-one, during the first call to readNext, you get one structure for each stream as expected.

data1 = readNext(streamReader)

data1=2×1 struct array with fields:
    StreamIndex
    Data

In the next call to readNext, we only get items for stream 1.

data2 = readNext(streamReader)

data2 = struct with fields:
    StreamIndex: 1
           Data: [1×1 struct]

Note that although one of the streams has reached end of selection, readNext still returns the data items from the remaining streams. Similarly, the hasNext function will return true even if one of the streams in the selection has data available to read.

hasNext(streamReader)

ans = logical
   1