Parallel Computing Toolbox - Java object transfer from client to workers

My problem regards the usage of the Parallel Computing Toolbox functions, and in particular the variable transfer from client to workers when the "variable" is a Java object. I have experieced that, without multithreading, passing a Java object from a main script to a function is not a problem, while using (for instance) parfeval to implement multithreading, the object is not passed correctly resulting in the error "Attempt to reference field of non-structure array", because the object structure is lost. Something similar happens to me also trying to implement multithreading exploiting a parfor cycle, therefore I suspect it may be a generalized issue with the usage of Matlab multithreading tools. May I kindly have any clarification with this regards?

Answer 1

[Edric Ellis]

When you pass any MATLAB variables (including those that refer to Java objects) into parfor, parfeval, etc., then the contents of those variables need to be transmitted to a separate MATLAB process. The effect is the same as if you saved the variables to a .mat file and then loaded them. Not all MATLAB variables can be saved and loaded successfully - and those same variables will not function correctly with parfor, parfeval etc.

I recommend that you should construct such objects on the workers (i.e. inside parfor, parfeval, ...).

You might wish to use the Worker Object Wrapper to avoid constructing the same object multiple times on the workers.

[Stefano C]

Many thanks for your answer Edric, unfortunately WorkerObjWrapper does not seem to work for me, in fact having this situation:

main.m

[...]
wrapperData = WorkerObjWrapper(@generateDataset, fName, numFolds);
for fold = 1 : numFolds    
   % call Parallel Function Evaluation
   CV(fold) = parfeval(p, @CVloop, 2, wrapperData.Value, numFolds, fold);
end

for fold = 1 : numFolds
     % fetchNext blocks until next results are available.
87:  [completedCV,value] = fetchNext(CV);
     fprintf('Got result #%d\n', completedCV);
end
[...]

CVloop.m

[...]
23: trainSet = randData.trainCV(numFolds, fold - 1);
[...]

I still get the error:

Error using parallel.FevalFuture/fetchNext (line 243)
The function evaluation completed with an error.

Error in main (line 87)
        [completedCV,value] = fetchNext(CV);

Caused by:
    Error using CVloop (line 23)
    Attempt to reference field of non-structure array.

Depite of that, using dedicated functions to create the Java objects for each worker, as you suggested, everything seems to work fine, even if not very efficiently (but acceptably).

Thanks again

[Edric Ellis]

You should call

CV(fold) = parfeval(p, @CVloop, 2, wrapperData, numFolds, fold);

to send the whole wrapper to the worker, and then only on the worker pick out the .Value field. (And just to be sure - you should be creating the WorkerObjWrapper and calling parfeval at the client).

[Stefano C]

Thanks a lot Edric, sending the whole wrapper I now get a different error:

Error using WorkerObjWrapper
Too many input arguments.

Error in main (line 62)
    wrapperData = WorkerObjWrapper(@generateDataset, WPATH, fName, numFolds);

It seems there is a sort of issue calling the wrapped function with multiple arguments, do you know why?

EDIT:

I have been able to solve this last question looking at the code of WorkerObjWrapper:

%    W = WORKEROBJWRAPPER( FCN, ARGSCELL ) creates a WorkerObjWrapper by
%    invoking FCN inside an SPMD block, supplying ARGSCELL as
%    arguments. I.e., W.Value has the value obtained from running
%    FCN(ARGSCELL{:}) on the workers.

Hence, for the posterity, in case the wrapped function has multiple inputs, WorkerObjWrapper has to be called in this way:

FcnArgs = {arg1, arg2, arg3, ...};
wrapperData = WorkerObjWrapper(@Fcn, FcnArgs);

[Stefano C]

Excuse me Edric, despite having (I think) correctly instantiated the objects wrappers on the client and picked out the .Value on the workers, still WorkerObjWrapper does not seem to work, I now get the error:

Error in main (line 90)
        [completedFold,value] = fetchNext(CV);

Caused by:
    Error using WorkerObjWrapper/get.Value (line 109)
    Assertion failed.

kindly, would you know what it may depend upon?