I need to build a tool that will allow access to extremely large data sets where the user will typically only work with smaller slices at a time. I know newer versions of MATLAB have tools built-in that can help deal with some of these issues, but I need to be backwards compatable with fairly old versions of MATLAB. The data is binary and can be stored across multiple files (approx 2 GB chunks). I need to expose this to the user as though they were all combined together and behave like a normal 3-dimensional vector which can be arbitrarily sliced.
Currently, I return a struct to the user that contains a handle to a function in place of the actual data matrix when the data set is too large for memory. This function is designed to be called like a normal vector slicing operation, and in return, it pieces together the desired components from the memory maps and returns an array to the user.
I would like to put something together that behaves more like an array without the need for the function call when its not necessary. For example, if the user were to use the size command, I'd rather it just return the size instead of actually having to build the thing to then figure out the size. Or, if the user tries to print the array at the prompt, they get an indication that it's a function handle, rather than a preview of the data set. Same thing with the preview in the variable list. Additionally, the user cannot use the end keyword when trying to build a slice, though I am able to use a plain colon operator. As an aside, if the user is trying to get a larger slice, they then have to pass an entire vector identifying the slice to the function call that has to be generated in memory. It would be nice for this to not happen.
So, where would I need to start to think about doing something like this? Does this need to be done in MEX, or can it be done in MATLAB code? I'm open to any thoughts in general.
