Error using GPU in Matlab 2020a for Reinforcement Learning

I keep running into this error when using 'UseDevice',"gpu" in rlRepresentationOptions. The issue seems to appear after the the simulation happens for random period of time. I have tried this with multiple built-in examples and with both DDPG and TD3 agent. Could someone direct me if I am doing something wrong or is this a bug?

Error using rl.env.AbstractEnv/simWithPolicy (line 70)
An error occurred while simulating "IntegratedFlyingRobot" with the agent "agent".

Error in rl.task.SeriesTrainTask/runImpl (line 33)
            [varargout{1},varargout{2}] = simWithPolicy(this.Env,this.Agent,simOpts);

Error in rl.task.Task/run (line 21)
            [varargout{1:nargout}] = runImpl(this);

Error in rl.task.TaskSpec/internal_run (line 159)
            [varargout{1:nargout}] = run(task);

Error in rl.task.TaskSpec/runDirect (line 163)
            [this.Outputs{1:getNumOutputs(this)}] = internal_run(this);

Error in rl.task.TaskSpec/runScalarTask (line 187)
                runDirect(this);

Error in rl.task.TaskSpec/run (line 69)
                runScalarTask(task);

Error in rl.train.SeriesTrainer/run (line 24)
            run(seriestaskspec);

Error in rl.train.TrainingManager/train (line 291)
            run(trainer);

Error in rl.train.TrainingManager/run (line 160)
            train(this);

Error in rl.agent.AbstractAgent/train (line 54)
TrainingStatistics = run(trainMgr);

Caused by:
    Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)
    Invalid input argument type or size such as observation, reward, isdone or loggedSignals.
        Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)
        Unable to compute gradient from representation.
            Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)
            Unable to evaluate the loss function. Check the loss function and ensure it runs successfully.
                Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)
                Input data dimensions must match the dimensions specified in the corresponding observation and action info specifications.

[Victor Antony]

Same Problem, please send help! This is insane.

[Daniel Egan]

I am also having the same problem, TD3 and DDPG agents, 2020a, training on a GPU (1080Ti), and this problem occurs even when I feed the RL Agent block values from constant blocks rather than from my dynamic model. See picture below for setup.

The model will successfully work through ~5 episodes before this same error pops up.

[Anh Tran]

We identified this is a bug in DDPG and TD3 with GPU training in R2020a. We are working on creating a patch to resolve this issue. As a workaround, please use CPU for DDPG and TD3.

[Stav Bar-Sheshet]

Found a workaround to use until this bug will get solved.

You can still train the Actor with GPU and the Critic with CPU.

With this configuration you can also still use the parallel pool for gathering multiple experiences faster.

This reduced my training time in comparison to just train with CPU for both A & C.

[Jing Chen]

I have a similar error using DDPG and cpu, but I can not understand what the bug is in the last line means. My obeservation and action are like the example in Train DDPG Agent to Swing Up and Balance Pendulum with Image Observation ( https://www.mathworks.com/help/deeplearning/ug/train-ddpg-agent-to-swing-up-and-balance-pendulum-with-image-observation.html?s_tid=srchtitle ). The environment is built in Simulink and the RLAgent in Simulink is used . While this example does not explain how to build an environment by ourselves.

Error in rl.task.SeriesTrainTask/runImpl (line 33)

[varargout{1},varargout{2}] = simWithPolicy(this.Env,this.Agent,simOpts);

Error in rl.task.Task/run (line 21)

[varargout{1:nargout}] = runImpl(this);

Error in rl.task.TaskSpec/internal_run (line 159)

[varargout{1:nargout}] = run(task);

Error in rl.task.TaskSpec/runDirect (line 163)

[this.Outputs{1:getNumOutputs(this)}] = internal_run(this);

Error in rl.task.TaskSpec/runScalarTask (line 187)

runDirect(this);

Error in rl.task.TaskSpec/run (line 69)

runScalarTask(task);

Error in rl.train.SeriesTrainer/run (line 24)

run(seriestaskspec)

Error in rl.train.TrainingManager/train (line 291)

run(trainer);

Error in rl.train.TrainingManager/run (line 160)

train(this);

Error in rl.agent.AbstractAgent/train (line 54)

TrainingStatistics = run(trainMgr);

Caused by:

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)

Invalid input argument type or size such as observation, reward, isdone or loggedSignals.

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)

Unable to compute gradient from representation.

Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 689)

Number of elements must not change. Use [] as one of the size inputs to automatically calculate the appropriate size for that dimension.

Answer 1

[Stephan]

If you install Update 2 the problem should be solved:

https://de.mathworks.com/support/bugreports/?release=R2020a&release_filter=Fixed+in&fir[]=R2020a-Update-2&product[]=RL

[Adriano Mele]

Hi Stephan,

I have a similar error, but my matlab version is 2021 and up-to-date (?)

I tried to build a simulink environment for RL DDPG training. If I simulate the environment alone (with zero actions) it seems to work just fine, but whenever I try to create an integrated environment and train the agent the code breaks with the same error reported above. I tried with both cpu and gpu.

Any hint?

[kloner]

I have a similar error on 2021a when considering tabular Q-Learning and observations larger than [1 x 1]. Guidance for this particular issue or more precise error statements would be greatly appreciated!