synchronised or asynchronised reinforcement learning

Dear reader,

I have a question regarding the use of:

trainOpts.ParallelizationOptions.Mode = 'sync' OR 'async'

As I understand from this and this, in synchronised learning the agent sends the data to each worker and waits for each worker to be finnished simulating in the environment in order to update the parameters, hence the agent will wait for the slowest worker to finnish.

In asynchronised learning (as applied in the quadruped robot and human robot), the agent is continuesly updating the data stream to each individual worker.

When is it advised to use synchronised over asynchronised learning? In my particular case, the number of episodes per hour seem to be a lot faster for the synchronised learning, does this mean it would get a solution faster as well, or will it take longer as the agent is updated less frequently?

I look forward to an answer.

Thank you,

GZ

Answer 1

[Gagan Agarwal]

Hi Gerlof Zuidema

When it comes to determining the use of Synchronized over Asynchronized learning, it is advisable to employ this approach when there are interdependencies between data and shared parameters.

For the reinforcement learning model, the number of episodes completed per hour is not the only factor that determines finding a solution. Other factors, such as the time taken for each update in the data stream, also play a role. Therefore, it is advisable to experiment with both approaches and determine the most suitable one for your specific use case.

I hope this answers your question!