Do we need a delay block in reinforcement learning simulink?

Hi

I have started using sumulink recently to use reinforcement learning toolbox. In Matlab examples, there is not any delay unit but in my simulation I got loop error and it will be fixed after I use delay block. The training can be excuted now but the learning is not succesful and I am wondering whether this delay is the reason or not because it is not used in matlab examples. I have screenshot of before and after using delay block. Errors were shown when there is no delay.

Caused by:

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Error due to multiple causes.

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input data dependency violation due to action subsystems. See Subsystem Examples in the

Simulink library for valid and invalid examples of action subsystems

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Calculate Reward/Abs' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2) of 'MODEL2D/Calculate Reward/If1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Calculate Reward/Abs1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem1' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2, 3) of 'MODEL2D/Calculate Reward/Merge' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem2' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem3' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/UAVEnv/Add' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Calculate Reward/Subtract1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Calculate Reward/Gain1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/UAVEnv/Add1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2) of 'MODEL2D/Calculate Reward/If' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2) of 'MODEL2D/Calculate Reward/Merge1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem4' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Abs1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant2/Compare' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Subtract1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant/Compare' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Abs' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant1/Compare' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2) of 'MODEL2D/Check if Done/Logical Operator' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Subtract' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant3/Compare' are involved in

the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2, 3) of 'MODEL2D/Check if Done/Logical Operator1' are involved in the

loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2, 3) of 'MODEL2D/RL Agent/AgentWrapper' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/UAVEnv/Sin' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2) of 'MODEL2D/UAVEnv/Product' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2, 3, 4) of 'MODEL2D/Calculate Reward/Add2' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/UAVEnv/Cos' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1, 2) of 'MODEL2D/UAVEnv/Product1' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Calculate Reward/Subtract' are involved in the loop

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)

Input ports (1) of 'MODEL2D/Calculate Reward/Gain' are involved in the loop

Answer 1

[Emmanouil Tzorakoleftherakis]

A delay block may be necessary to break algebraic loops or to rely on observations from previous time steps (see e.g. this example). I would try to identify which specific observation causes the issue - no need to add a delay to all of them.

[SAMir]

Thanks Emmanouil. I fixed it somehow but still have strange result. Please have a look on my other question:

https://www.mathworks.com/matlabcentral/answers/513368-deep-deterministic-policy-gradient-agents-ddpg-at-reinforcement-learning-actor-output-is-oscilat