Large Binary Data Files: Can I Animate while Simultaneously Loading the Next Data Block Into Memory from Disk?

Hi There,

I'm making tools for visualizing very large data sets. A typical routine will often look something like:

function AnimateFile(fidIn,BytesPerStep,NumSteps,StepsPerBlock,GraphicsHandles)

NumBlocks = ceil(NumSteps/StepsPerBlock);
for BlockNum = 1:NumBlocks
    DataBlock = fread(fidIn,[BytesPerStep StepsPerBlock],'*uint8');
    for StepNum = 1:size(DataBlock,2)
        MakePrettyPictures(DataBlock(:,StepNum),GraphicsHandles)
    end
end

But of course this means that the animation runs in fits and starts as it jumps back and forth between rendering in the inner loop and waiting for the fread() in the outer loop to complete. Is there some way of making the fread() command run "in the background" to load up the data for the next block while the current block is being animated?

-Jan

Answer 1

[Oleg Komarov]

EDIT

1. Read in first block
2. Start timer which executes the TimerFcn after the previous execution has ended (so no hard coded delays). The TimerFcn now doesn't share the same workspace so it should be enough to pass DataBlock as an argument to make one single copy
3. In the meantime the import should proceed overwriting the DataBlock with the condition that TimerFcn is working on the previous

Let me know if it works.

function AnimateFile(fidIn,BytesPerStep,NumSteps,StepsPerBlock,GraphicsHandles)

NumBlocks = ceil(NumSteps/StepsPerBlock);

% Read in first block
DataBlock = fread(fidIn,[BytesPerStep StepsPerBlock],'*uint8');

% Create timer object which executes mpp every .5 seconds with a delay of
% .5 second to wait for the first chunk of data to be loaded
t = timer('ExecutionMode' , 'fixedSpacing',...
          'Period'        , 0            ,...
          'BusyMode'      , 'queue'       ,...
          'TimerFcn'      ,{@mpp,DataBlock,GraphicsHandles});
start(t);

for b = 2:NumBlocks
    % Cannot load the third block if still executing the first 
    % Example: if b = 3 and TaskExecuted = 1 then it can proceed otherways wait
    while b - get(t,'TasksExecuted') > 2 && strcmp(get(t,'Running'),'on')
        pause(0.1) 
    end
    % Overwrite
    DataBlock = fread(fidIn,[BytesPerStep StepsPerBlock],'*uint8');
end

end

%  Sub function
function mpp(varargin)
for StepNum = 1:size(varargin{3},2)
    MakePrettyPictures(varargin{3}(:,StepNum),varargin{4})
end

end

[Jan Rubak]

Hi Oleg,

Thanks for your response. I haven't used timers before, so I'll have to have a closer look at that option. My main constraint is that I'm dealing with very large data files, which is why I have to load them in one or two blocks at a time. Is there some way to delay the read from file operation for each data block until the animation of the previous data block commences (and to then delete each data block from memory once its animation has been carried out)?

Also, I'd prefer not to use hard-coded delays like 0.5, as the data load time will vary with BytesPerBlock and with the hardware. Thoughts?

Cheers,

Jan

[Oleg Komarov]

Edited above.

[Jan Rubak]

Hi Oleg,

Thank you for this. I got pulled off onto a different task, so I haven't had a chance to test this yet. I will report back after I do.

Cheers,

Jan

[Oleg Komarov]

Looking forward fro your feedback.