Error in rl.agent.A​bstractAge​nt/train while executing example "Train Reinforcement Learning Agent in MDP Environment"

Hi,

I'm trying to run an example from the Reinforcement learning Toolbox examples, but the function train() does not work correctly. The example is "Train Reinforcement Learning Agent in MDP Environment". I have only changed the value of doTraining to true to be able to execute that part of the code.

This is the code of the example:

MDP = createMDP(8,["up";"down"]);

% State 1 transition and reward

MDP.T(1,2,1) = 1;

MDP.R(1,2,1) = 3;

MDP.T(1,3,2) = 1;

MDP.R(1,3,2) = 1;

% State 2 transition and reward

MDP.T(2,4,1) = 1;

MDP.R(2,4,1) = 2;

MDP.T(2,5,2) = 1;

MDP.R(2,5,2) = 1;

% State 3 transition and reward

MDP.T(3,5,1) = 1;

MDP.R(3,5,1) = 2;

MDP.T(3,6,2) = 1;

MDP.R(3,6,2) = 4;

% State 4 transition and reward

MDP.T(4,7,1) = 1;

MDP.R(4,7,1) = 3;

MDP.T(4,8,2) = 1;

MDP.R(4,8,2) = 2;

% State 5 transition and reward

MDP.T(5,7,1) = 1;

MDP.R(5,7,1) = 1;

MDP.T(5,8,2) = 1;

MDP.R(5,8,2) = 9;

% State 6 transition and reward

MDP.T(6,7,1) = 1;

MDP.R(6,7,1) = 5;

MDP.T(6,8,2) = 1;

MDP.R(6,8,2) = 1;

% State 7 transition and reward

MDP.T(7,7,1) = 1;

MDP.R(7,7,1) = 0;

MDP.T(7,7,2) = 1;

MDP.R(7,7,2) = 0;

% State 8 transition and reward

MDP.T(8,8,1) = 1;

MDP.R(8,8,1) = 0;

MDP.T(8,8,2) = 1;

MDP.R(8,8,2) = 0;

MDP.TerminalStates = ["s7";"s8"];

env = rlMDPEnv(MDP);

env.ResetFcn = @() 1;

rng(0);

obsInfo = getObservationInfo(env);

actInfo = getActionInfo(env);

qTable = rlTable(obsInfo, actInfo);

qFunction = rlQValueFunction(qTable, obsInfo, actInfo);

qOptions = rlOptimizerOptions("LearnRate",1);

agentOpts = rlQAgentOptions;

agentOpts.DiscountFactor = 1;

agentOpts.EpsilonGreedyExploration.Epsilon = 0.9;

agentOpts.EpsilonGreedyExploration.EpsilonDecay = 0.01;

agentOpts.CriticOptimizerOptions = qOptions;

qAgent = rlQAgent(qFunction,agentOpts); %#ok<NASGU>

trainOpts = rlTrainingOptions;

trainOpts.MaxStepsPerEpisode = 50;

trainOpts.MaxEpisodes = 500;

trainOpts.StopTrainingCriteria = "AverageReward";

trainOpts.StopTrainingValue = 13;

trainOpts.ScoreAveragingWindowLength = 30;

doTraining = true;

if doTraining

% Train the agent.

trainingStats = train(qAgent,env,trainOpts); %#ok<UNRCH>

else

% Load pretrained agent for the example.

load('genericMDPQAgent.mat','qAgent');

end

Data = sim(qAgent,env);

And this is the error:

Error using rl.train.SeriesTrainer/run

There was an error executing the ProcessExperienceFcn.

Caused by:

Error using rl.function.AbstractFunction/gradient

Unable to compute gradient from function model.

Error in rl.agent.rlQAgent/learn_ (line 228)

CriticGradient = gradient(this.Critic_,lossFcn,...

Error in rl.agent.AbstractAgent/learn (line 29)

this = learn_(this,experience);

Error in rl.util.agentProcessStepExperience (line 6)

learn(Agent,Exp);

Error in rl.env.internal.FunctionHandlePolicyExperienceProcessor/processExperience_ (line 31)

[this.Policy_,this.Data_] = feval(this.Fcn_,...

Error in rl.env.internal.ExperienceProcessorInterface/processExperienceInternal_ (line 137)

processExperience_(this,experience,getEpisodeInfoData(this));

Error in rl.env.internal.ExperienceProcessorInterface/processExperience (line 78)

stopsim = processExperienceInternal_(this,experience,simTime);

Error in rl.env.internal.MATLABSimulator/simInternal_ (line 128)

stopsim = processExperience(expProcessor,exp,i*ts);

Error in rl.env.internal.MATLABSimulator/sim_ (line 67)

out = simInternal_(this,simPkg);

Error in rl.env.internal.AbstractSimulator/sim (line 30)

out = sim_(this,simData,policy,processExpFcn,processExpData);

Error in rl.env.AbstractEnv/runEpisode (line 144)

out = sim(simulator,simData,policy,processExpFcn,processExpData);

Error in rl.train.SeriesTrainer/run (line 32)

out = runEpisode(...

Error in rl.train.TrainingManager/train (line 429)

run(trainer);

Error in rl.train.TrainingManager/run (line 218)

train(this);

Error in rl.agent.AbstractAgent/train (line 83)

trainingResult = run(trainMgr,checkpoint);

Error in test_IA (line 76)

trainingStats = train(qAgent,env,trainOpts); %#ok<UNRCH>

Caused by:

Undefined function 'struct2array' for input arguments of type 'struct'.

Error in rl.train.TrainingManager/train (line 429)

run(trainer);

Error in rl.train.TrainingManager/run (line 218)

train(this);

Error in rl.agent.AbstractAgent/train (line 83)

trainingResult = run(trainMgr,checkpoint);

Error in test_IA (line 76)

trainingStats = train(qAgent,env,trainOpts); %#ok<UNRCH>

[Gunawan]

Any solution for this problem? I also have the same error.

[David Carrasco Martínez]

I think the train function only works in Matlab Online. I dont know why, but the example finally worked in the online version.

Answer 1

[Yichen]

I meet the same problem !

[Gunawan]

Hello, have you resolved this error? how to overcome this error? thanks.

Answer 2

[MULI]

Hi @David Carrasco Martínez,

I understand that you are facing an issue when “doTraining” flag is set to “true” in "Train Reinforcement Learning Agent in MDP Environment" example.

You can try the below suggestions to resolve this issue:

• Check whether you have “signal processing toolbox” and “DSP system toolbox” installed.
• Also Check if the“struct2array”function is available in your MATLAB installation. You can do this by typing:

>>which struct2array 

You can refer to the below MATLAB answer to get some additional insights on this issue:

https://www.mathworks.com/matlabcentral/answers/493612-any-fixes-for-undefined-function-or-variable-struct2array