Steepest descents methods algoritme for higher dimensional objective functions

Hello,

I am trying to apply the steepest descent method on a function with 10 variables.

With 2 variables it is easy as I can split the problem. Now I tried to write this algoritme for a vector, but without succes.

In the following code you will first see a simple steepest descent algorithm and in the code below you see a similar algoritme based on a vector as input, especially needed to tackle higher dimensional problems, by using the vector notation in this algoritm.

Can I have some feedback.

Clarisha

clc
close
clear

%objective function
b=@(x,y) (1-x).^2+(y-x.^2).^2 

%de gradient
dbdx=@(x,y) (2-4*x)-4*x*(y-x.^2)
dbdy=@(x,y)  2*(y-x.^2)

%initials
x0=20
y0=20

%proces
for i=1:10
  s1=dbdx(x0,y0);
  s2=dbdy(x0,y0);

  xd=@(d) x0+d*s1;
  yd=@(d) y0+d*s2;

  
  bd=@(d)  b(xd(d),yd(d));
  d_star=fminsearch(bd,0)
  
  x1=xd(d_star); 
  y1=yd(d_star) ;

  iteratie=i
  x0=x1%update initials
  y0=y1%update initials
  ObjectiveValue=b(x0,y0)
end


%********************************************************************************************************
%Steepest descent method for functions with more input.
B=@(X) (1-X).^2;
DBDX=@(X) -2*(1-X);
X0=[0 0]; %initials

for iteration=1:N
    S=DBDX(X0);
    XK=@(D) X0+D.*S;
    BK=@(D) B(XK(D));
    D_STAR=fminsearch(BK,X0);
    X=XK(D_STAR)
    X0=X
end

[Matt J]

Can I have some feedback.

My feedback would be that you should not use steepest descent for high dimensional problems. It is one of the slowest possible algorithms. Is there a reason you are not using the more advanced algorithms of the Optimization Toolbox?

[Clarisha]

Thank you for your quick response. I am actually trying to solve a PDE constraint problem with numerical methods. So I splitted the problem in two and one of the parts is optimizing an unconstrainted multidimensional function. As I am not so experienced in programming, I tried the steepest descent problem first. But what other methods would you suggest?

[Clarisha]

For your information I also tried the conjugate gradient method and a gmres code. And chose for the one I could best interprete in matlab.

Answer 1

[Matt J]

See fminunc, fmincon, and perhaps also,

https://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html

[Clarisha]

The ODE part is not a problem. I solved that numerically. But the optimal cost function should be solved now for several variables simultaneously.

[Matt J]

Yes, fmincon and fminunc are multi-variable solvers.

No, the fact that you've solved the differential equations numerically inside your objective function does not mean you have overcome the issues described at,

https://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html

You need to read it carefully.

[Clarisha]

Thank you so much Matt J; I will study this link tomorrow.

[Clarisha]

Thank you for the references. I just skipped the steepest descent method and used the fiminunc().

[Matt J]

You're welcome, but please Accept-click the answer to indicate that it resolved your question.

Answer 2

[Walter Roberson]

fminsearch() uses simplex algorithm, not Steepest Descent.

One implementation of Steepest Descent is https://www.mathworks.com/matlabcentral/answers/787539-steepest-descent-algorithm-in-matlab#answer_1191330

[Matt J]

But I think here fminsearch is being used to do the 1D line search step of steepest descent.

If so, it probably would have been better to use fminbnd.

[Clarisha]

Yes I read somewhere that the fminbind () is used for 1 dimensional problems. But infact I have a multivariate fcostunction with more than 30 dimensions (variables) and I am looking for the combination of these 30 variables that optimizes my costfunction.