Direct GPU-to-GPU Communication with Parallel Computing Toolbox / SPMD

I am using spmd to enable parallel computing with multiple GPUs on one workstation. Basically, the GPUs do some calculation, broadcast their results, update their parameters, and iterate. The problem is, using labSend (actually, gplus in my case) to aggregate and broadcast the results is pretty slow. It is first pulling the results off of the GPU, copying to system memory, sending to other workers, then uploading to the other GPUs.

I understand that CUDA now has Peer-to-Peer memory access capability. This way, multiple-GPUs can directly access each other's memory. http://www.nvidia.com/docs/IO/116711/sc11-multi-gpu.pdf This is accomplished with a function like: cudaMemcpyPeerAsync().

Thus, I would like to have a gplus() or labSend() that copies a gpuArray directly to the memory of another GPU on another worker.

Is this possible today? If not, is it something you are working on?

Thanks, Jon

Answer 1

[Edric Ellis]

Unfortunately, as you observe, Parallel Computing Toolbox currently has no means by which to achieve this. I believe you can use the peer-to-peer memory copying across multiple processes within a single node, which means you could use the GPU MEX interface to copy data.

[Jonathan]

Basically, what I am trying to accomplish is to execute the same function (with different data) on two GPUs simultaneously. Then I sum the results, update some parameters and repeat.

I assume, I could write a MEX function that takes as input a matlab function handle to evaluate, and multiple gpuArrays to operate on. It would return the sum of the results in two different gpuArrays (each array having identical values, but stored on different GPUs), having performed the calculation on multiple GPUs. The problem is, can you have gpuArrays pointing to data stored on different GPUs in a single worker or client?

It seems like the problem here is that, outside of the MEX function, regular matlab scripting cannot handle gpuArrays pointing to different devices, no?