Custom DDPG Algorithm in MATLAB R2023b: Performing Gradient Ascent for Actor Network

Hello MATLAB community,

I am working on implementing a custom Deep Deterministic Policy Gradients (DDPG) algorithm in MATLAB R2023b. In the DDPG algorithm, during the training of the actor network, the Q value produced by the critic network is set as the objective function for the actor network. The standard approach involves using gradient ascent to update the actor network based on these Q values.

My question pertains to the use of the gradient function from the Reinforcement Learning Toolbox to calculate gradients. Following this, how can I perform gradient ascent, as the update function from the same toolbox seems to default to gradient descent and not gradient ascent? I would appreciate any insights or examples on implementing gradient ascent in this context.

Thank you for your assistance!

Answer 1

[Venu]

Hi @roham farhadi,

Gradient ascent is the same as gradient descent except that you don't multiply your step (learning_rate * gradients) by a negative sign. So your step has the same sign as your gradient.

If the update function defaults to gradient descent, you can adjust the sign of the gradients before updating the parameters.

actorNetwork.Parameters = actorNetwork.Parameters + learningRate * -gradients; (% Perform gradient ascent by adjusting the sign of the gradients)

You can refer to example in this documentation for 'gradient' function

https://www.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-policy-using-custom-training.html,

Hope this helps!