FMINCON runs with lower and upper bound variables switched, but doesn't run when bounds are specified correctly

I'm running fmincon function in python via this interface: https://github.com/byuflowlab/pyfmincon

If I specify upper and lower bounds as

ub = np.array([1.0, 30.0])
lb = np.array([0, 0])

it results in the following error:

Unable to resolve the name 'py.self.neg_log_likelihood'.

Error in my_optimize/fupdate (line 45)
        eval(['output = py.', funcname, '(x);'])  % output is a cell array with {J, cin}

Error in my_optimize/obj (line 63)
        [J, cin, gJ, gcin] = fupdate(x);

Error in fmincon (line 577)
      [initVals.f,initVals.g] = feval(funfcn{3},X,varargin{:});

Error in my_optimize (line 28)
    [xopt, fopt, exitflag, output] = fmincon(@obj, x0, A, b, Aeq, beq, lb, ub, @con, options);

Caused by:
    Failure in initial objective function evaluation. FMINCON cannot continue.

However, if I switch upper and lower bounds the function runs without errors and prints out this message:

    Exiting due to infeasibility: at least one lower bound exceeds the corresponding upper bound.

[Walter Roberson]

Do not use eval. Use

output = py.(funcname)(x);

[Anastasia Lado]

Thanks! But unfortunately it didn't solve the problem.

I get a feeling that this error: "Unable to resolve the name 'py.self.neg_log_likelihood'." is the most important one. Do you know how to solve this one?

[Walter Roberson]

wait, does funcname contain a period? It is not obvious to me that "self" should be part of the name.

[Anastasia Lado]

this funcrion is a part of the class, so self. is necessary

[Walter Roberson]

My suggestion to use dynamic field names would need to be modified if the supplied name includes periods. So for the moment go back to eval() version

[Anastasia Lado]

it doesn't work with eval as well

Answer 1

[Walter Roberson]

Let us examine how fmincon works.

The first thing that fmincon does is validate its parameters as being the correct data type and consistent sizes.

The next thing it does is to invoke the objective function passing in x0, in a section of code that has extra checks to see if the function returns a reasonable value -- in particular a finite numeric scalar.

Then it involves the objective function once for each variable, with a modified version of the initial conditions, in order to get locations and values that can be used to estimate the gradient. During this initial pass, nonlinear constraints are ignored.

Okay so as mentioned first it validates data types. What is permitted for the initial location, x0? https://www.mathworks.com/help/optim/ug/fmincon.html#d123e91785 Answer: real-valued double precision vector. If you had passed in initial values that the py interface was not able to convert to double precision, then if the interface did not catch it, then fmincon would catch it when checking the parameters.

Now the objective function is invoked with a double precision vector. The objective function is supplied by the py interface. It takes the double precision vector and passes it to a Python function, relying on the usual conversions of double precision to the appropriate Python data type. The Python function receives double precision, possibly np.array or something like that.

Now, in order for self to work, the recieved class (np.array?) would have to have the target method neg_log_likelihood. Does it? And if it does, then is it really necessary to invoke self.neg_log_likelihood specifically or would it figure it out based on the object passed in?

If you look at the example you will see that the example passes in a static method name as the function, not 'self.something'

[Anastasia Lado]

the class has the neg_log_likelihood method. it works well with other optimization functions. here is the definition:

class ML(object):
    def __init__(self, df, n_actions, cues=None, bounds=None,
        model='constant_step_size',
        param_value={'alpha':None, 'beta':None, 'r_bar':None}):
        """The DataFrame df must contain columns 'action' and reward'.
        If `len(cues) > 1`, then it also must include the 'cue' column.
        model can be 'sample_average', 'constant_step_size', 'policy', or
        'policy_daw'.
        """
        if model not in ('sample_average', 'constant_step_size', 'policy',
        'policy_daw'):
        raise ValueError("model must be 'sample_average' or "
        "'constant_step_size'")
        self.n_actions = n_actions
        self.model = model
        if cues is None:
            if 'cue' in df.columns:
                #self.cues = (df['cue'].values[0],)
                self.cues = (df['cue'].unique().tolist())
                print('Using {} for the cue.'.format(self.cues[0]))
            else:
                self.cues = (0,)
                df['cue'] = 0
            else:
                if type(cues) is not tuple:
                    raise TypeError('cues must be a tuple')
                    self.cues = cues
                if type(cues) is int:
                    self.cues = (cues,)
                    self.df = df
                    self.bounds = bounds
                    # Initial conditions passed to the optimization function
                if self.model in ('sample_average', 'constant_step_size'):
                    self.opt_ic = [0.1, 2.0]
                    elif (self.model == 'policy') or (self.model == 'policy_daw'):
                    self.param_value = param_value
                    n_unknow = list(param_value.values()).count(None)
                if n_unknow == 0:
                    raise ValueError('At least one unknown parameter is needed')
                    self.opt_ic = np.random.uniform(0.1, 0.2, n_unknow)
                    print(f'ML estimation model:{model} param:{param_value}\n')
                            
    def neg_log_likelihood(self, params):
    """Compute the negative log likelihood of the parameters.
    The data consist of a sequence of (cue,action,reward) observations.
    """
    df = self.df
    if self.model in ('sample_average', 'constant_step_size'):
    alpha, beta = params
    elif (self.model == 'policy') or (self.model == 'policy_daw'):
        if self.param_value['alpha'] == None:
            alpha, params = params[0], params[1:]
        else:
        alpha = self.param_value['alpha']
        if self.param_value['beta'] == None:
            beta, params = params[0], params[1:]
        else:
            beta = self.param_value['beta']
        if self.param_value['r_bar'] == None:
            r_bar, params = params[0], params[1:]
        else:
            r_bar = self.param_value['r_bar']
        
    df = self.df[self.df['cue'].isin(self.cues)]
    actions, rewards = df['action'].values, df['reward'].values
    cues = df['cue'].values
    prob_log = 0.0
    Q = dict([[cue, np.zeros(self.n_actions)] for cue in self.cues])
    k = 1
    for action, reward, cue in zip(actions, rewards, cues):
        if action == 'blue':
            action = 1
        else:
            action = 0
        if self.model == 'sample_average':
            Q[cue][action] += alpha * (reward - Q[cue][action]) / k
            k += 1
        elif self.model == 'constant_step_size':
            Q[cue][action] = Q[cue][action] + alpha * (reward - Q[cue][action])
            #Q[cue] = Q[cue].at[action].set((Q[cue][action] + alpha * (reward - Q[cue][action])))
        elif self.model == 'policy':
            # We are reusing Q, but the right name should be pi
            probs = softmax(Q[cue], beta)
            for a in (0,1):  # (0, 1) should be something like self.actions
                indicator = 1 if a == action else 0
                Q[cue][a] += alpha * (reward - r_bar) * (indicator - probs[a])
        elif self.model == 'policy_daw': # Daw simplified rule
            # We are reusing Q, but the right name should be pi
            Q[cue][action] += alpha * (reward - r_bar)
        else:
            print('Something bad happen :(')
            prob_log += np.log(softmax_jnp(Q[cue], beta)[action])
                
       return -prob_log

[Walter Roberson]

In order for that to be of any use, Python would have to be acting on an object of Python class ML. Let us look at

https://github.com/byuflowlab/pyfmincon/blob/master/opt.py

eng = matlab.engine.start_matlab(start_options)

    # convert to numpy array then list then to matlab type
    # these first conversions are necessary to allow both numpy and list style inputs
    x0 = matlab.double(np.array(x0).tolist())
    ub = matlab.double(np.array(ub).tolist())
    lb = matlab.double(np.array(lb).tolist())
    A = matlab.double(np.array(A).tolist())
    b = matlab.double(np.array(b).tolist())
    Aeq = matlab.double(np.array(Aeq).tolist())
    beq = matlab.double(np.array(beq).tolist())

    # run fmincon
    print('--- calling fmincon ---')
    [xopt, fopt, exitflag, output] = eng.optimize(function, x0, A, b,
        Aeq, beq, lb, ub, options, providegradients, nargout=4)

Notice that every parameter passed to eng.optimize is matlab.double except for the character vector in the variable "double". Therefore MATLAB is not being passed any Python class objects and therefore no objects for which self might have a method neg_log_likelihood.

[Walter Roberson]

What is the engine asked to execute?

https://github.com/byuflowlab/pyfmincon/blob/master/optimize.m

That builds an options structure, and calls MATLAB fmincon with the matlab.double passed in, and the options structure. The MATLAB fmincon that is called is passed matlab @obj and MATLAB function obj that is provided in the above link calls MATLAB function fupdate that is provided in the above link. So MATLAB fmincon was passed MATLAB double precision and will invoke obj on MATLAB double precision which will invoke MATLAB fupdate on MATLAB double precision. fupdate is the one that has

 eval(['output = py.', funcname, '(x);'])  % output is a cell array with {J, cin}

where in scope x refers to the MATLAB double precision passed into MATLAB fupdate.

This is the point at which you want self.neg_log_likelihood to refer to an object in the Python-level class ML. But clearly it does not, and cannot

[Walter Roberson]

In order to get this to work the Python level function you pass in would have to refer to a Python level global variable of Python class ML and use the methods of that Python-level variable to update the mf content, unless what is being optimized over is alpha and beta in which case the logic becomes easier.

[Anastasia Lado]

I want to optimize over alpha and beta.

Did I understand you correctly: the function I want to optimize shouldn't be a part of the class.

(by the way, I really appreciate your help and the time you put into it)

[Walter Roberson]

The function you want to optimize should either not be part of a class or else should be part of whatever class that matlab double precision variables get received at inside Python.

You can create a function that accesses a global variable if class ML and invokes neg_log_likekihood(OBJECT, x) where x is the value passed in by MATLAB

[Anastasia Lado]

I hope I understood you correclty: I added function that invokes matlab into the same class as the function I'm optimizing, and the error is gone.

However, it's replaced by a new error (ML is my class):

  File "/home/anastasia/RL OIST/contextual_bandit_2.py", line 323, in fmincon
    [xopt, fopt, exitflag, output] = eng.my_optimize(function, x0, A, b,

  File "/home/anastasia/anaconda3/lib/python3.8/site-packages/matlab/engine/matlabengine.py", line 64, in __call__
    future = pythonengine.evaluateFunction(self._engine()._matlab,

TypeError: unsupported Python data type: __main__.ML

[Walter Roberson]

I speculate that you somehow made x0 into an ML object instead of using the previous interface that would have forced it to be matlab.double

[Anastasia Lado]

I put arrays within that function (so I don't pass it to function anymore) and it solved the issue, but now I'm getting the error message that looks in part like the old one (fupdate is the function within the pyfmincon package, optimize.m file):

Unable to resolve the name 'py.self.neg_log_likelihood()'.

Error in my_optimize/fupdate (line 44)
        output = py.(funcname)(x);

Error in my_optimize/obj (line 62)
        [J, cin, gJ, gcin] = fupdate(x);

Error in fmincon (line 577)
      [initVals.f,initVals.g] = feval(funfcn{3},X,varargin{:});

Error in my_optimize (line 26)
    [xopt, fopt, exitflag, output] = fmincon(@obj, x0, A, b, Aeq, beq, lb, ub, @con, options);

Caused by:
    Failure in initial objective function evaluation. FMINCON cannot continue.

Error using my_optimize/fupdate
Error:  Invalid array indexing.

Error in my_optimize/obj (line 62)
        [J, cin, gJ, gcin] = fupdate(x);

Error in fmincon (line 577)
      [initVals.f,initVals.g] = feval(funfcn{3},X,varargin{:});

Error in my_optimize (line 26)
    [xopt, fopt, exitflag, output] = fmincon(@obj, x0, A, b, Aeq, beq, lb, ub, @con, options);

Caused by:
    Failure in initial objective function evaluation. FMINCON cannot continue.

[Walter Roberson]

As I explained before, for there to be any chance for self to be valid in the name of the function to execute, then it would have to be the case that self referred to whatever datatype a MATLAB double precision number gets converted to when it is passed back to Python.

You are constructing this wrong. Do not make the call part of the ML class. Define a different function such as

ML_stored_for_optimization = 0;   %just need to create the variable outside of scope
def optimize_ML_ab(ML_Object)
    ML_stored_for_optimization = ML_Object;
    bestab = fmincon('calc_nll', ... stuff);
    return bestab;
def calc_nll(params)
    cost = ML_stored_for_optimization.neg_log_likelihood(params);
    return cost;

This does not require any change to the class, and does not require that MATLAB be passed the Python ML object. It does require that the non-method calc_nll has access to a stored ML object to invoke the neg_log_likelihood of.

[Anastasia Lado]

I made this functions not part of the class. They looks like this now:

ML_stored_for_optimization = 0
def calc_nll():
    cost = ML_stored_for_optimization.neg_log_likelihood([0.5, 5])
    return cost

def optimize_ML_ab(ML_Object):
    global ML_stored_for_optimization #this is added to make sure that this variable is modified
    ML_stored_for_optimization = ML_Object
    bestab = fmincon('calc_nll')
    return bestab

And the call:

ml = ML(df, 2, model=model,
        param_value={'alpha':None, 'beta':None, 'r_bar':None})
optimize_ML_ab(ml) 

And I'm still getting the same error:

Unable to resolve the name 'py.calc_nll'.

Error in my_optimize/fupdate (line 43)
        eval(['output = py.', funcname, '(x);'])  % output is a cell array with {J, cin}

Error in my_optimize/obj (line 62)
        [J, cin, gJ, gcin] = fupdate(x);

Error in fmincon (line 577)
      [initVals.f,initVals.g] = feval(funfcn{3},X,varargin{:});

Error in my_optimize (line 26)
    [xopt, fopt, exitflag, output] = fmincon(@obj, x0, A, b, Aeq, beq, lb, ub, @con, options);

Caused by:
    Failure in initial objective function evaluation. FMINCON cannot continue.

[Walter Roberson]

My guess is that you need to supply the name of the .py file as part of the function name, such as

    bestab = fmincon('my_optimize.calc_nll');
    

Note that calc_nll needs to receive params and pass them to ML_stored_for_optimization.neg_log_likelihood as otherwise you would be trying to optimize something that is computationally constant.

[Anastasia Lado]

Thanks a lot! Your suggestion worked.

My only problem now is that it runs for 6 hours and then my computer crashes. Is it because I didn't pass any parameters? Would you be able to help me with that since I don't know how to pass parameters to a function that is in quotes ''.

[Anastasia Lado]

I tried replacing my function with a simplier one, and also tried printing out a line every time the objective function is called. But the message is never printed out, the code is running and crashing my computer. So it looks like the code never enters calc_nll(), because when i added a print message there at the very beginning it was never shown.