How can I access fmincon's function count within a custom anonymous function?

I am trying to use fmincon to optimise an aerofoil shape with CFD (computational fluid dynamics). I have defined my own anonymous function to generate the aerofoil shape and run the CFD case on a remote cluster. I want to access fmincon's iteration number and function count within my function so I can name the input files to track the algorithm's progress. This is particularly important when I run fmincon in parallel as any nasty workarounds (global variables or writing files to disc) fail horribly and you end up with files with the same name.

I see that the build in plotting functions like optimplotfval must know the iteration number but I can't work out how to use them in my case.

If you can't access the function count directly, manually indexing the calls to the function in some other robust way would be fine.

I have included an example below, please can anyone help me?

    function [x,fval] = optimise_aerofoil()
    % Calculate input parameters (x) that yield the best aerofoil design 
    % measured by the output (fval)

    % Define some constants that are needed to run the case
    a = 1; b = 2; c = 3;

    % Initialise the first design and bounds
    x0 = [1 2 3 4 5]; lb = [0 0 0 0 0]; ub = [10 10 10 10 10];

    % Define the anonymous function handle to run the cfd and return 
    % the output value
    f = @(x)run_cfd(x,a,b,c);

    % Choose optimiser settings
    options = optimoptions('fmincon','Display','iter','UseParallel',1);

    % Run the optimiser
    [x,fval] = fmincon(f,x0,[],[],[],[],lb,ub,[],options);    

    end

    function f = run_cfd(x,a,b,c)
    % Generate a design, save it, run the CFD and evaluate the design

    % Generate the design from x
    xyz = generate_design(x,a,b,c);

    %%%%%%%HERE IS WHERE I WOULD LIKE TO KNOW F-COUNT %%%%%%%
    iter_count = ?;

    % Save the input file to CFD
    filename = ['Run_' num2str(iter_count) '.dat'];
    dlmwrite(filename,xyz);

    % Run the CFD
    run_cfd(filename)

    % Evaluate the design
    f = evaluate_design(filename);

    end

[Matt J]

I don't really see how the code you've posted can run without error messages. The run_cfd() function is multiply defined - both as your objective function to fmincon and as some external routine

    run_cfd(filename)

that takes filename as input.

[James]

Sorry, I made a mistake in calling both functions run_cfd in my example. I should have called the second one something else.

    ...
    dlmwrite(filename,xyz);

    % Run the CFD on a remote machine
    run_remote_cfd_case(filename)

    % Evaluate the design
    ...

Answer 1

[Steven Lord]

Use an output function. According to the documentation that lists the fields passed into the OutputFcn, fmincon does pass the field "iteration" into the OutputFcn in the optimValues struct array.

[James]

Thank you for your help.

Isn't the output function only called once and at the end of the iteration? How can I get the function call count inside the actual function (run_cfd) which is being used within the fmincon?

[Matt J]

The output function is called at several points throughout the iteration (including the end), as explained in States of the Algorithm.

[James]

One call to fmincon contains maybe 10 iterations, each iteration within fmincon contains maybe 10 calls to my function run_cfd. I can't see how the output function will allow me to access data about the number of function calls within my function run_cfd?

[Steven Lord]

I'm not sure if this will work in your parallel use case, but maybe the memoize function introduced in release R2017a can help you in reusing previously computed results.

[Matt J]

@James,

It appears that you have two purposes here. One purpose is to generate unique filenames to avoid conflicts between parallel calls to run_remote_cfd_case(). I believe this is already covered by suggestions made by Walter and me.

The second purpose is to save the results of each iteration so that you can resume the optimization if it aborts prematurely. The latter is best done using an Output Function as Steve suggests, here. There is an example of how to do this in the documentation on Output Functions, but basically, it would look like this for you

     % Choose optimiser settings
    options = optimoptions('fmincon','Display','iter',...
                           'UseParallel',1,'OutputFcn',@outfun);

    function stop = outfun(x,optimValues,state)

       if strcmp(state,'iter')

          filename = ['Iteration_' num2str(optimValues.iteration)];

          save(filename,'x');

       end

    end

Answer 2

[Walter Roberson]

Perhaps you could use tempname() to generate unique names? Though I am not certain that parallel workers have JVM

With R2017a onward, you might be able to use the new send() / poll() facilities.

[James]

Thank you, this is a nice idea.

Ideally I'd like to have names which are repeatable though. Running the CFD is quite expensive and sometimes tempermental. If I restart fmincon I'd like to to recognise that it has already converged some designs and re-use the output.

[Matt J]

If I restart fmincon I'd like to to recognise that it has already converged some designs and re-use the output.

For that purpose, it is a bad idea to save all evaluations of your objective function. Some evaluations result in no progress, or worse. You should be saving the final result of each iteration, using the output function method suggested by Steve, and as demonstrated in this example.

Answer 3

[Matt J]

It might make the most sense to generate a temporary filename based on labindex()

   filename = ['Run_' num2str(labindex) '.dat'];

The iteration count and funccount will not let you distinguish between parallel workers, as labindex does.

[Walter Roberson]

"However, inside a parfor-loop, labindex always returns a value of 1 on all workers in all iterations."

[Matt J]

OK, but this will work in parfor

   t=getCurrentTask(); t.ID;
 filename = ['Run_' num2str(t.ID) '.dat'];

For some reason, though, Edric Ellis does prefer tempname()