Why does Soft actor critic have Entropy terms instead of Log probability?

According to the Soft Actor Critic paper by Haarnoja et al. (2018) the TD learning, Policy update and the entropy coefficient or temperature update all have used log probability inside the Expectation symbol due to the soft state function and thus leading to Entropy indirectly. I want to know if it is a documentation error in MATLAB 2021a that entropy was used directly or is there a coding error in the backend. SInce i cant seem to find the function for the training loops for these functions i cannot verify for myself. I will put the formulas for comparison here as images as it might exceed the characters.

From the Spinning up in RL Documentation by Open AI parent company of the authors who tweaked the algo slightly to include only Q values, here only Log probability is used and then summed over.

In MATLAB's Documentation we have entropy before the summation

SAC is a very important off policy reinforcement learning algorithm for various research purposes which specialises in sample efficiency, if the mistakes in the documentation is reflected in the code then it will be a term higher in degree than entropy and thus non accurate results will occur and if it is a minor documentation error, nevertheless it needs to be fixed.

Answer 1

[Takeshi Takahashi]

RL toolbox also uses the log of the probability density to approximate the differential entropy.

[Aniruddha Datta]

Yes that's okay but the thing is ... Entropy is not used as the soft update in the Q function in the original paper, log probability is used and then there is an Expectation Operation which "Results" in Entropy ....I guess we don't start with Entropy in the TD learning step.And that also doesn't explain why there is a Squared term in the entropy coefficient Alpha optimization.

[Takeshi Takahashi]

The squared term in the entropy coefficient is a typo. The update equation is the same as the paper. Thank you for pointing out this.

SAC is derived from the maximum entropy RL framework that has a (differential) entropy term in its objective function. I believe that's how the log density is included in the TD learning step, and it is called the entropy term. It is also true that it results in entropy if it starts from the log density.

Policy update is derived from KL-divergence, and the KL-divergence can be viewed as relative entropy, so it has an entropy of the policy. KL(policy||exp(Q)) = -H(policy) + H(policy,exp(Q)).

[Aniruddha Datta]

So in effect , differential entropy was not mentioned in the documentation?

[Aniruddha Datta]

Or was it that the log density was just called Entropy in the documentation.

[Aniruddha Datta]

Is it possible to look at the source code for the training loop of the policy and Critic?

[Takeshi Takahashi]

Yes. You are right that we call the log density Entropy in our SAC documentation.

Answer 2

[Aniruddha Datta]

The follow up paper, Soft Actor Critic Algorithm and Applications is much more consistent in the terms used for Soft Q update and Soft policy iteration.

[Takeshi Takahashi]

Thank you for your feedback!