Peformance issue while fetching data from C Mex function.

Hi,

We have C mex funciton that is implemented to fetch data from external source over network and fill this data in C mex function(using mxCreateDoubleMatrix and copy this data into plhs array), when this data is huge(around 22 MB), accessing this data on matlab is very slow takes while(20 seconds), if we chunk this data(in 100Kbs), data is copied and accessed on matlab variable faster. I wanted to know, is there memory limit while allocating and copying data and access into matlab variable. Appreciate your suggestions.

Thank you'

[Jan]

Without seeing the code it is impossible to guess, if it contains an avoidable bottleneck. 22MB is not considered as "huge" usually. "100Kbs" is a strange unit and it is not clear, how this could define a chunk.

The memory limit is defined in Matlab's preferences => WorkSpace => Array size limit

[Manju gurik]

Thank you for the reply. here is the code.

Mex Function: ReadRows.c

void mexFunction( int nlhs, mxArray *plhs[],

int nrhs, const mxArray *prhs[] )

{

size_t cnt, num;

int OutputIndex = 0;

double ***pFields, **pTmp;

double ConnNumber, ReadIndex, NumFields, NumtoRead, *InputFieldSize, MaxtoRead;

double *ReadStatus, *NumRead, **OutputFieldSize, **BitFieldStatus;

size_t iFields, iRead;

ConnNumber = mxGetScalar(prhs[0]);

ReadIndex = mxGetScalar(prhs[1]);

NumFields = mxGetScalar(prhs[2]);

InputFieldSize = mxGetPr(prhs[3]);

NumtoRead = (int)mxGetScalar(prhs[4]);

plhs[OutputIndex] = mxCreateDoubleMatrix(1, 1, mxREAL);

ReadStatus = mxGetPr(plhs[OutputIndex++]); /* Read status */

*ReadStatus = 1; // ADI_STAT_FAILURE

plhs[OutputIndex] = mxCreateDoubleMatrix(1, 1, mxREAL);

NumRead = mxGetPr(plhs[OutputIndex++]); /* Number Read */

*NumRead = 0;

iFields = (size_t)NumFields;

NumFields = iFields;

// limit to left hand side inputs

MaxtoRead = (size_t)( ((double)nlhs-2.0) / (NumFields+2.0) );

if ( NumtoRead > MaxtoRead )

NumtoRead = MaxtoRead;

QueryNumberOfRows( ConnNumber, &MaxtoRead );

if ( NumtoRead > MaxtoRead )

NumtoRead = MaxtoRead;

iRead = (size_t)NumtoRead;

NumtoRead = iRead;

if ( nlhs < 2+NumtoRead*(NumFields+2) )

mexErrMsgTxt("Need outputs: Status, NumRecsRead, [BitStat, OutputSize, NumFieldsOfData...]*NumtoRead");

if ( NumtoRead > 0 )

{

BitFieldStatus = (double**)mxMalloc( (mwSize)2*iRead*sizeof(double*) );

OutputFieldSize = BitFieldStatus + iRead;

pFields = (double***)mxMalloc((mwSize)iRead*sizeof(double**) );

// since iFields usually small compared to iRead, allocate block

pTmp = (double**)mxMalloc((mwSize)iRead*iFields*sizeof(double*) );

}

for(num = 0; num < (size_t)NumtoRead; num++)

{

/* Output arguments */

plhs[OutputIndex] = mxCreateDoubleMatrix((mwSize)iFields, 1, mxREAL);

BitFieldStatus[num] = mxGetPr(plhs[OutputIndex++]); /* Read bit field status */

plhs[OutputIndex] = mxCreateDoubleMatrix((mwSize)iFields, 1, mxREAL);

OutputFieldSize[num] = mxGetPr(plhs[OutputIndex++]); /* Field size read */

// pFields[num] = (double**)mxMalloc( iFields*sizeof(double*) );

pFields[num] = pTmp + num*iFields;

/* Output Field arguments */

for(cnt = 0; cnt < NumFields; cnt++)

{

plhs[OutputIndex] = mxCreateDoubleMatrix((mwSize)InputFieldSize[cnt], 1, mxREAL);

pFields[num][cnt] = mxGetPr(plhs[OutputIndex++]);

}

}

while( OutputIndex < nlhs )

{

plhs[OutputIndex++] = mxCreateDoubleMatrix(0, 0, mxREAL);

}

FetchData(ConnNumber, ReadIndex, NumFields, NumtoRead, InputFieldSize, ReadStatus, NumRead, BitFieldStatus, OutputFieldSize, pFields);

}

Matlab Calling:

chunkSize = 50000;

num_cols = NumFields + 2; %(3 data fields + 2(bitfieldstatus and outputsize)

for k = 0:chunkSize:TotalNumRows

startIndex = k;

dataStartIndex = startIndex + 1;

dataEndIndex = startIndex + chunkSize;

if (dataEndIndex > TotalNumRecords)

dataEndIndex = TotalNumRecords;

chunkSize = TotalNumRecords - startIndex; % Last chunk remaining records that are less than chunk size(2048)

clear varargout;

end

[varargout{1:(NumFields + 2)*chunkSize+2}] = ...

ReadRows(conNum, startIndex, NumFields, InputFieldSize, ...

chunkSize);

data_A(dataStartIndex:dataEndIndex) = [varargout{1, 5:num_cols:end}]';

data_B(dataStartIndex:dataEndIndex) = [varargout{1, 6:num_cols:end}]';

data_C(dataStartIndex:dataEndIndex) = [varargout{1, 7:num_cols:end}]';

end

In the above code, when we use chunkSize = 2048, data assignments to data_A, data_B, data_C is immediate, if we use chunkSize = 50000, it will take a while(30 seconds) to reach to this line

data_A(dataStartIndex:dataEndIndex) = [varargout{1, 5:num_cols:end}]';

even though mex function returns immediatly afte fetching data from FetchData() which takes less than a second.

My question, why its taking time to copy/transfer the data after mex function ReadRows call when we have bigger chunks. Is there any way to improve this.

Here all the data fields are double values(vector).

Thank you.