Prealllocation of undefined file size

Hi, I have question. Does preallocation must be perform on a defined size? Now I have a situation that I m receiving bits from serial port. I do not know large is the file size but i will keep receiving it with this codes:

received_count=1;

file=[];

received=fgetl(s);

file=[file received];

received_count=received_count+1;

And will stored inside the file variable until I fully received the bits in a large array let say (1,M) but M is unknown values. So how do i preallocate this file to minimize the execution time?

Answer 1

[Walter Roberson]

I already answered this for you.

http://www.mathworks.com/matlabcentral/answers/6127-fgetl-fscanf-fgets-fprintf-execution-time#comment_12766

As that comment is now hidden in the Older Comments list, I will repeat it here:

You can preallocate to your maximum expected size, or you can dynamically allocate in chunks (e.g., 16 Kb) so as to reduce the number of allocations you need to do. To allocate in chunks, you would determine whether the new data you have would fit within the existing buffer, and if not then extend the buffer by a number of elements and then write the new data in to it. for example, file(end+16384)=0 .

[MrPuzzled Marc]

Is it like this one?

file=zeros(16,100000); %Where 100000 is the maximum expected size

received=fgetl(s);

file(:,end)=received;

Is this what you mean?

[Walter Roberson]

I'm not sure why you are working with 16 bits at a time instead of 8, but anyhow...

file=zeros(16,100000); %Where 100000 is the maximum expected size

curpos = 0;

while true

received=fgetl(s);

if ~ischar(received); break; end %end of file

t = mod(length(received),16); %might not have received full group

if ~t; received(end+16-t) = '0'; end %if not, pad to full group

ncols = length(received)/16;

file(:,curpos+1:curpos+ncols) = reshape(received,16,ncols);

curpos = curpos + ncols;

end

file(:,curpos+1:end) = []; %remove unused part of buffer

and remember to transpose "file" before doing bin2dec on it.

In the above code, curpos has been written to reflect the amount of data that has already been received rather than the next position available to be written in to.

If your lines might get split in the middle of the group and you want to rejoin the group across lines, then you are better off reading everything in to vector instead of in to a matrix.

[Walter Roberson]

The version of this with dynamic allocation is like this:

%use a larger ChunkSize for higher efficiency

%use a smaller ChunkSize to have less overallocation at the end

ChunkSize = 16384;

file = zeros(16, ChunkSize);

curpos = 0;

while true

received = fgetl(s);

if ~ischar(received); break; end %end of file

t = mod(length(received),16); %might not have received full group

if ~t; received(end+16-t) = '0'; end %if not, pad to full group

ncols = length(received)/16;

if curpos + ncols > size(file,2) %past end of buffer

file(end,end+ChunkSize) = 0; %extend buffer

end

file(:,curpos+1:curpos+ncols) = reshape(received,16,ncols);

curpos = curpos + ncols;

end

file(:,curpos+1:end) = []; %remove unused part of buffer

and remember to transpose "file" before doing bin2dec on it.

In the above code, curpos has been written to reflect the amount of data that has already been received rather than the next position available to be written in to.

If your lines might get split in the middle of the group and you want to rejoin the group across lines, then you are better off reading everything in to vector instead of in to a matrix.

[Walter Roberson]

By the way, with the dynamic allocation scheme, you should expect that there will still be a warning that the array may be growing within the loop. That's because it _is_ growing within the loop. It is not an error to grow an array within a loop: it just causes efficiency problems. The above code reduces the inefficiency.

There are dynamic allocation schemes that are more efficient than the above, that use cell arrays to assemble the data, but the code for those is more complex.

[MrPuzzled Marc]

I had tried out with the code you given to me however, error occur. How can i solve this?The data obtained from fgetl is char.

??? Error using ==> reshape

Size arguments must be real integers.

Error in ==> vlc2>Receive_Callback at 326

file(:,curpos+1:curpos+ncols) = reshape(received,16,ncols);