How can I listen for completion of a job?

I'm working with a very large and complex GUI that is used to process hundreds of files at a time. Many of these files can be processed independently, and each file may take minutes to process, making them excellent candidates for parallel processing.

I would like to update the GUI whenever a file is done processing. The standard way of getting results from parallel processing is using fetchOutputs or fetchNext. However, these functions block continuation, thus interfering with my ability to interact with the GUI.

It seems to me that the best way of doing this would be to simply listen for completion of each Job, Task, or Future. However, the only events these objects seem to have is ObjectBeingDestroyed. They have a State property which can be used to determine whether or not they have been completed, but this property is not SetObservable.

I could poll the State property (or poll fetchNext with a TimeOut) using a loop or timer and notify when it is 'finished', but I expect that will be difficult to manage, and will occupy resourced on the MATLAB Client.

So is there any way to listen for completion of a job?

Answer 1

[Lionel]

OK, it looks like I found an answer using functions included in version 2017a.

% First define a parallel pool 
poolobj = parpool('local',1);
% then create a data queue
q = parallel.pool.DataQueue;
% create a listener on that queue
listener = afterEach(q, @testevent);
% and start the data processing (note that the queue must be passed)
f = parfeval(@testfun, 0, data, q);

The function testfun must accept the queue as an argument and triggers the event by using send:

function testfun(data,q)
    % process the data here
    send(q,dataprocessed)
end

Once the data is sent back, it triggers the function testevent and the data is given as an argument:

function testevent(dataprocessed)
    % perform some actions

One can use this in conjunction with local workspaces and handles in order to store the result directly into the workspace. To do this, replace the line:

listener = afterEach(q, @testevent);

by:

listener = afterEach(q, makeListenerHandle(someHandle));

where the function makeListenerHandle generates a handle to a nested function:

function functionHandle = makeListenerHandle(handleStructure)
    functionHandle = @LocalFunction;
    function LocalFunction(dataprocessed)
      handleStructure.dataprocessed = dataprocessed;
    end
end

I made a few scripts and it is working well on version 2017a, using windows 10.

[Yongjun Taylor]

Lionel, your solution is the most valuable! I've been writing a code that runs parfeval in a timer callback to guarantee a rough periodic real time execution. I was stuck on fetching the result automatically. Now your answer help me a lot! Thank you for sharing your hard work.

[Juan Miguel Serrano Rodríguez]

Thank you very much Lionel

Answer 2

[Edric Ellis]

Unfortunately, there's currently no way to get event notifications from the parallel.Future objects returned by parfeval.

Under certain circumstances (i.e. only for MJS cluster types), there are various callbacks available: https://www.mathworks.com/help/distcomp/apply-callbacks-to-mjs-jobs-and-tasks.html.

[Alexander]

Thank you for your help. I was afraid that was true. I might look into MJS clusters in the future, but configuring for an MJS seems more complicated polling at the current development stage.

Answer 3

[Walter Roberson]

parfeval(). That will create futures. You can check the State property of the futures without waiting for them to complete.

[Alexander]

Thanks for your input.

However, as I mentioned at the end of my question, checking the State property requires polling, which occupies resources on the Client and is substantially more complicated than listening for a completion event. I would only use this technique if there is no way to listen for completion of the job.

Answer 4

[Lionel]

Hi, Sorry to revive an old post but I am interested in the exact same issue: how to trigger an event on completion of a parfeval task. I see that parfeval objects have an 'ObjectBeingDestroyed' event which could be useful. Is there a way to automatically destroy a parfeval object when its task is complete? If this is the case, then I could set an event just before the object is destroyed and collect the data.