Parfor waitbar : How to do this more cleanly?

I'm a frequent casual user of parfor loops. Getting a waitbar (progress meter) from parallel code used to involve having to have each thread append to a file to record progress, and there are a number of contributions on the file exchange that do this. It was kinda silly, but it was necessary.

Now that the DataQueue system exists, it should be possible to do this more sensibly. So I went looking for a good waitbar on the File Exchange. And I couldn't find one, so I tried to make my own.

The code below works nicely, but it depends on using three globals - which isn't great, because if those variable names are being used by something else, it's gonna break stuff. Can anybody suggest a better way to do this, preferably not involving globals? Persistent variables should work within update.m, but it's getting the info (waitbar handle and expected number of interations) from create.m to update.m that's stumped me.

There are three functions, all part of the ParWaitbar package. Usage is explained in the first one:

create.m:

function [ q ] = create( n, txt )
%FNPARWAITBAR Produces a waitbar object that works with parfor.
% Relies on globals. Not sure how to avoid this.
%   Input:  
%           n: Number of iterations expected
%           txt: Text for the waitbar dialog.
%   Output: q: handle for the dataqueue
%
%   Usage:  [ q ] = ParWaitbar.create( 10, "Predicting earthquakes..." );
%           parfor i=1:10
%               pause(rand*5); (or do useful stuff)
%               send( q, i ); % doesn't matter what's sent (i here). We're just
%                               counting the items rcvd.
%           end
%           ParWaitbar.cleanup(q);

global wb; %waitbar handle.
global wb_N; % number of iterations completed
global wb_max; %number of iterations expected

q = parallel.pool.DataQueue;
wb = waitbar( 0, txt );
wb_max = n;
wb_N = 0;

afterEach( q, @ParWaitbar.update );

end

update.m:

function update(~)

global wb; %shoudl have waitbar handle.
global wb_N; % number of iterations completed
global wb_max; %number of iterations expected

wb_N = wb_N + 1;
waitbar( wb_N / wb_max, wb );

end

cleanup.m

function cleanup( q )
% FIXME we're left with q afterwards. Presumably a fn can't delete
% something outside its own scope

global wb; %shoudl have waitbar handle.
global wb_N; % number of iterations completed
global wb_max; %number of iterations expected

close(wb);
delete(q);
clear wb wb_N wb_max q

end

(as an aside, the fact that we're left with q leftover as a "handle to deleted DataQueue" isn't ideal. But that's far less of a problem than the use of globals with fixed names.)

Answer 1

[Edric Ellis]

Hm, I've been meaning to tidy up my work-in-progress parallel.pool.DataQueue pool waitbar for quite a while. Here's roughly what it would look like. This uses a handle object that gets copied to the workers, so there's no need for global state.

classdef PoolWaitbar < handle
    properties (SetAccess = immutable, GetAccess = private)
        Queue
        N
    end
    properties (Access = private, Transient)
        ClientHandle = []
        Count = 0
    end
    properties (SetAccess = immutable, GetAccess = private, Transient)
        Listener = []
    end

    methods (Access = private)
        function localIncrement(obj)
            obj.Count = 1 + obj.Count;
            waitbar(obj.Count / obj.N, obj.ClientHandle);
        end
    end
    methods
        function obj = PoolWaitbar(N, message)
            if nargin < 2
                message = 'PoolWaitbar';
            end
            obj.N = N;
            obj.ClientHandle = waitbar(0, message);
            obj.Queue = parallel.pool.DataQueue;
            obj.Listener = afterEach(obj.Queue, @(~) localIncrement(obj));
        end
        function increment(obj)
            send(obj.Queue, true);
        end
        function delete(obj)
            delete(obj.ClientHandle);
            delete(obj.Queue);
        end
    end
end

This works with parfor, spmd, and parfeval, like this:

pw = PoolWaitbar(100, 'Example');
parfor ii = 1:20
    increment(pw)
end
spmd
    for ii = 21:40
        if labindex == 1
            increment(pw);
        end
    end
end
for ii = 41:100
    parfeval(@() increment(pw), 0);
end

[Simon]

Fabulous, thank you! I don't fully understand Matlab's OO facilities, but this seems to do nicely.

Is there any reason to do increment(pw) instead of pw.increment? The latter feels more intuitive to me for using an object's method, but I don't know what normal is in Matlab for this

Are you planning to put this on File Exchange?

[Edric Ellis]

This PoolWaitbar does use a slightly subtle trick to make things work - the use of Transient properties stops the client-side handle being sent to the workers.

There's no real difference between increment(pw) and pw.increment - but the former is the usual function-call style that tends to be used by MathWorks documentation and example code.

I would like to put this on File Exchange - but it's part of a larger suite of similar utilities, and there's some polishing required...