Agent repeats same sequence of actions each episode
조회 수: 7 (최근 30일)
이전 댓글 표시
Braydon Westmoreland
2020년 7월 1일
편집: Emmanouil Tzorakoleftherakis
2020년 7월 2일
Can someone please help me understand why my RL Agent is outputting the same sequence of actions each episode, regardless of the observations made from the environment. Here is an example of what I mean:
prev_state = 11.20 11.90 11.30 11.50
action = 0.00 0.00 0.00 0.00
new_state = 11.20 11.90 11.30 11.50
prev_state = 11.20 11.90 11.30 11.50
action = 0.10 0.10 -0.10 0.00
new_state = 11.30 12.00 11.20 11.50
prev_state = 11.30 12.00 11.20 11.50
action = 0.10 0.10 -0.10 0.00
new_state = 11.40 12.00 11.10 11.50
prev_state = 11.40 12.00 11.10 11.50
action = -0.10 -0.10 0.10 0.00
new_state = 11.30 11.90 11.20 11.50
prev_state = 11.30 11.90 11.20 11.50
action = 0.00 0.00 0.10 0.10
new_state = 11.30 11.90 11.30 11.60
Episode: 1/ 2 | Episode Reward : -5.00 | Episode Steps: 5 | Avg Reward : -5.00 | Step Count : 5 | Episode Q0 : 1.03
prev_state = 12.00 11.20 11.70 11.50
action = 0.00 0.00 0.00 0.00
new_state = 12.00 11.20 11.70 11.50
prev_state = 12.00 11.20 11.70 11.50
action = 0.10 0.10 -0.10 0.00
new_state = 12.00 11.30 11.60 11.50
prev_state = 12.00 11.30 11.60 11.50
action = 0.10 0.10 -0.10 0.00
new_state = 12.00 11.40 11.50 11.50
prev_state = 12.00 11.40 11.50 11.50
action = -0.10 -0.10 0.10 0.00
new_state = 11.90 11.30 11.60 11.50
prev_state = 11.90 11.30 11.60 11.50
action = 0.00 0.00 0.10 0.10
new_state = 11.90 11.30 11.70 11.60
Episode: 2/ 2 | Episode Reward : -5.00 | Episode Steps: 5 | Avg Reward : -5.00 | Step Count : 10 | Episode Q0 : 1.04
Let me know if you have any questions about the simulation.
More info on the simulation & my other issues: https://www.mathworks.com/matlabcentral/answers/555799-reinforcement-learning-sample-time
댓글 수: 0
채택된 답변
Emmanouil Tzorakoleftherakis
2020년 7월 2일
편집: Emmanouil Tzorakoleftherakis
2020년 7월 2일
Hi Braydon,
I am not really sure why you are only looking at the first two episodes. RL can take thousands of episodes to converge so the first few really don't give you enough information. As a matter of fact, I ran your models for 20 episodes and the action sequence was different after a few episodes or so. If nothing else, I would check the reward formulation since this would drive how the neural networks weights change and thus how actions are selected (in addition to exploration).
Episode: 17/ 20 | Episode Reward : -5.00 | Episode Steps: 5 | Avg Reward : -5.00 | Step Count : 85 | Episode Q0 : -120.83
1.0000e-04
prev_state = 11.90 11.90 12.00 11.20
action = 0.00 0.00 0.00 0.00
new_state = 11.90 11.90 12.00 11.20
prev_state = 11.90 11.90 12.00 11.20
action = 0.10 0.10 -0.10 0.00
new_state = 12.00 12.00 11.90 11.20
prev_state = 12.00 12.00 11.90 11.20
action = -0.10 0.00 -0.10 0.10
new_state = 11.90 12.00 11.80 11.30
prev_state = 11.90 12.00 11.80 11.30
action = -0.10 0.10 0.00 -0.10
new_state = 11.80 12.00 11.80 11.20
prev_state = 11.80 12.00 11.80 11.20
action = 0.10 0.00 -0.10 0.00
new_state = 11.90 12.00 11.70 11.20
Episode: 18/ 20 | Episode Reward : -5.00 | Episode Steps: 5 | Avg Reward : -5.00 | Step Count : 90 | Episode Q0 : -83.15
1.0000e-04
prev_state = 11.70 11.90 11.50 11.60
action = 0.00 0.00 0.00 0.00
new_state = 11.70 11.90 11.50 11.60
prev_state = 11.70 11.90 11.50 11.60
action = 0.10 0.10 -0.10 0.00
new_state = 11.80 12.00 11.40 11.60
prev_state = 11.80 12.00 11.40 11.60
action = -0.10 0.00 -0.10 0.10
new_state = 11.70 12.00 11.30 11.70
prev_state = 11.70 12.00 11.30 11.70
action = -0.10 0.10 0.00 -0.10
new_state = 11.60 12.00 11.30 11.60
prev_state = 11.60 12.00 11.30 11.60
action = 0.10 0.00 -0.10 0.00
new_state = 11.70 12.00 11.20 11.60
댓글 수: 0
추가 답변 (0개)
참고 항목
카테고리
Help Center 및 File Exchange에서 Training and Simulation에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!