Parpool local too slow

Why does it take so long to start a parallel pool on a local profile? It's only 10 seconds or so, but for a job that should simply thread across the cores of the host machine's CPU it should be nearly instantaneous.

I spent some time reworking a bit of code we use frequently to run in parallel. The runtime went from 10 seconds to around 2 which I was very happy with. Our use case, though, only requires running a few jobs at a time. Over time it would add up to a pretty nice time savings except that every time the user runs the code they need to wait for parpool to run. The 15 seconds there means that it's only worth using the parallel code I worked so hard on if they're going to be running more than a handful of jobs at a time which is rare. The result is that my time producing parallel code was wasted because Matlab inexplicably needs time to set up a pool of workers on the host machine that Matlab itself is running on. Is there any way around this because it makes no sense?

[Matt J]

Are you re-opening the parpool everytime the code is re-run in the same MATLAB session? You know you don't have to, right? The parpool can be left open in the background as long as the MATLAB session lives.

[Adam]

I have also sometimes run into a situation where my code would be faster to parallelise if a pool is already open, but not worth doing so if I have to open the pool in order to do it. Thee things are all testable of course and, as Matt J, pointed out, you can leave the parallel pool open, but it can add needless complexity to code

[ali kiaee]

the long time is usually takes is to look for the license for every worker. I could get it working very fast by starting the Matlab session with -c argument and passing the license file location manually.

Answer 1

[Edric Ellis]

Unfortunately there is no way around this, other than to open a parpool when you first need one (or let it open automatically if you're using >= R2013b), and don't close it until you've done everything you need to do. In R2013b, there's also an automatic timeout so that the pool will shut down when you're not using it.

The reason for the amount of time taken is that the pool launches separate MATLAB worker processes (which are essentially the same as a normal MATLAB process, except they are missing the display), rather than simply creating additional threads.

[Carl]

That would be nice if this were improved. The users can just keep a parpool open but the use of the code is sporadic - it's hard to know when you start if you're going to need to run it 3 or 30 times, but 3 is more common so people mostly aren't using the parallel option.

Anyway, thanks for the thorough answer. At least now I'm sure there isn't another way to set this up.