Is it possible to create a sub-array without using additional memory on Matlab?

Well, I am trying to implement an algorithm on Matlab. It requires the usage of a slice of an high dimensional array inside a for loop. When I try to use the logical indexing, Matlab creates an additional copy of that slice and since my array is huge, it takes a lot of time.

 slice = x(startInd:endInd);

What I am trying to do is to use that slice without copying it. I just need the slice data to input a linear operator. I won't update that part during the iterations.

To do so, I tried to write a Mex file whose output is a double type array and whose size is equal to the intended slice data size.

 plhs[0] = mxCreateUninitNumericMatrix(0, 0, mxDOUBLE_CLASS,mxREAL); % initialize but do not allocate any additional memory
 ptr1 = mxGetPr(prhs[0]); % get the pointer of the input data

Then set the pointer of the output to the starting index of the input data.

 mxSetPr(plhs[0], ptr1+startInd);
 mxSetM(plhs[0], 1);
 mxSetN(plhs[0], (endInd-startInd)); % Update the dimensions as intended

When I set the starting index to be zero, it just works fine. When I try to give other values than 0, Mex file compiles with no error but Matlab crashes when the Mex function is called.

 slice = mex_slicer(x, startInd, endInd);

What might be the problem here?

ps: when doing logical indexing, it creates the index array of 'size = size([startInd:endInd])'. So, what I try to do here is a bit of hacking. Just give the pointer to a memory place where I know what exists and tell how many elements to obtain after that memory. I don't want to create any index array, too. (size of the index array in the order of 2 millions).

[Stephen23]

All of your examples show subscript indexing or linear indexing:

slice = x(startInd:endInd);

but none of them seem to show any logical indexing. The MATLAB documentation explains the difference:

https://www.mathworks.com/help/matlab/math/matrix-indexing.html

https://www.mathworks.com/company/newsletters/articles/matrix-indexing-in-matlab.html

Answer 1

[Jan]

Bruno Luong has implemented an inplace slice: See https://www.mathworks.com/matlabcentral/fileexchange/23576-min-max-selection, especially the function inplacesolumnmex.c . It worked fluently in my tests, but in the debug mode working with the temporary arrays let Matlab crash tremendously. Therefore I do not use it in productive code.

It might be easier to provide the complete array together with the indices:

slice = {x, startInd, endInd};

Then you x is created as shared data copy, such that no additional memory is used (except for the about 100 bytes for the header). Then the provided information should be enough to access the data inplace - as long as the contents of x is not changed. Of course you have to modify the functions to consider this type of object. But this should be less hard than fighting with Matlab's memory manager and the effects of shared data copies.

[Mehmet Fatih SAHIN]

Thanks for the answer. This is exactly what I am looking for. But the link is not to where you refer to.

See the following instead:

InplaceArray

Answer 2

[James Tursa]

What you are attempting to do creates an invalid mxArray. That is, the data pointers of an mxArray must be known by the MATLAB Memory Manager as being the initial elements of MATLAB allocated data memory blocks. They cannot be from the middle of such a block, and they cannot be from native C/C++ allocation functions (even at the start of such blocks). Attempting to create such an mxArray as you are doing will cause an assert seg fault on recent versions of MATLAB the moment you call the mxSetPr function (on older versions of MATLAB, the mxSetPr would allow the call, but the resulting mxArray is still invalid and would crash MATLAB later on).

Bruno Luong's FEX contribution that Jan Simon mentions hacks into the mxArray to avoid the assert seg fault, but the resulting mxArray is still invalid. So you have to be very, very careful how you use it and how you destroy it in order not to crash MATLAB. It can be done, but it is very tricky even for advanced users to get it right without crashing.

Best option, if possible, is to pass the indexes into the mex routine and do the processing all within the mex routine. That will allow you to read those data subsets without copying them, of course. The downside is you need to code the processing in the mex routine.

"... When I set the starting index to be zero, it just works fine ..."

No, it didn't. You essentially created a shared data copy of the original mxArray (two different mxArray's using the same data pointer(s)) in an illegal way. This will eventually crash MATLAB also because MATLAB doesn't officially know that these two mxArray's are sharing the same data pointer. The way to do it correctly in a mex routine is to use the following unofficial API routine:

mxArray *mxCreateSharedDataCopy(const mxArray *);

[James Tursa]

Another option that was just posted to the FEX for getting shared data copies of contiguous sub-sections of an existing variable:

https://www.mathworks.com/matlabcentral/fileexchange/65842-sharedchild-creates-a-shared-data-copy-of-a-contiguous-subsection-of-an-existing-variable