memmapfile system memory usage and transfer rate

Hi, I'm trying to use memory mapped files to transfer data between Matlab instances. The data isn't huge, only about 100MB. I noticed that when I'm writing, my memory usage goes up way more than the size of the data. (More than 1GB). It's also much slower than I expected, about 35MB/second. I have 4GB of physical memory and I haven't hit that limit yet so I don't think I'm paging. Sure seems like I'm still writing to disk. Anybody run into this issue or is it just me ?

Answer 1

[Walter Roberson]

Memory mapping uses the file system by definition.

If your instances are on the same node, then what you want is instead a shared memory segment. MATLAB does not support shared memory itself, but you can find MATLAB File Exchange contributions that handle it.

[Marty]

Thanks !

Answer 2

[Anthony Barone]

Are you using memory mapped .mat files? If so, and this is just a guess, but it could be because .mat files use compression (well, the v7 (default) and v7.3 ones do). This means that it takes some work to compress them while saving and while loading data (hense the extra memory usage and the slow response time). Unless your data would benefit from compression (example: it is mostly zeros), then saving and reading uncompressed data would probably be faster and use less memory.

Here is some information related to saving performance of different types of files (.mat and other): http://undocumentedmatlab.com/blog/improving-save-performance . It may be of interest to you for this issue.

[Walter Roberson]

I would think that .mat files cannot be memory mapped as anything other than byte streams, as they do not have binary structures that could be easily translated into structures and arrays.