Reinforcement learning problem. My robot doesn't move in Simulink

I'm trying that my robot goes to a concrete location via reinforcement learning. I have this written code. I don´t know if the problem is in a bad connection between the simulink scheme and the code. I attach the scheme.

% Crear interfaz de entorno

% Crea la especificación de observación.

numObs = 4; % x,y,θ,dθ

obsInfo = rlNumericSpec([numObs 1]);

obsInfo.Name = 'observations';

% Crea la especificación de acción.

numAct = 2;

actInfo = rlNumericSpec([numAct 1],'LowerLimit',-1,'UpperLimit',1);

actInfo.Name = 'foot_torque';

% ¿Por qué foot_torque?

% Crea la interfaz de entorno para el modelo de robot andante

mdl = 'robotdiferencial';

Ts= 0.025;

Tf = 10;

blk = [mdl,'/RL Agent'];

env = rlSimulinkEnv(mdl,blk,obsInfo,actInfo);

% env.ResetFcn = @(in) walkerResetFcn(in,upper_leg_length/100,lower_leg_length/100,h/100);

% Seleccionar y crear un agente para la formación

agent = createDDPGAgent(numObs,obsInfo,numAct,actInfo,Ts);

% Especificar las opciones de formación y el agente de formación

maxEpisodes = 2000;

maxSteps = floor(Tf/Ts);

trainOpts = rlTrainingOptions(...

'MaxEpisodes',maxEpisodes,...

'MaxStepsPerEpisode',maxSteps,...

'ScoreAveragingWindowLength',250,...

'Verbose',false,...

'Plots','training-progress',...

'StopTrainingCriteria','EpisodeCount',...

'StopTrainingValue',maxEpisodes,...

'SaveAgentCriteria','EpisodeCount',...

'SaveAgentValue',maxEpisodes);

trainOpts.UseParallel = true;

trainOpts.ParallelizationOptions.Mode = 'async';

trainOpts.ParallelizationOptions.StepsUntilDataIsSent = 32;

trainOpts.ParallelizationOptions.DataToSendFromWorkers = 'Experiences';

training = train(agent,env,trainOpts);

Answer 1

[Devesh Bhasin]

I tried running your code on my end. Are you encountering the error: 'createDDPGAgent' is used in Train Biped Robot to Walk Using Reinforcement Learning Agents.?

If so, this error occurs because the ‘createDDPGAgent’ is not a standard shipped function but is shipped as a part of the ‘Train Biped Robot to Walk Using Reinforcement Learning Agents’ example.

To resolve this error, follow the following steps:

1. Go to the example’s directory by clicking on the link in the error, or by going to the release-specific directory: ‘\MATLAB\Examples\R2022b\control_deeplearning\TrainBipedRobotToWalkUsingReinforcementLearningAgentsExample’.
2. Copy ‘createDDPGAgent.m’ and ‘CreateNetworks.m’ files. Paste them in your project directory. ‘CreateNetworks’ is a function used by ‘‘createDDPGAgent’.

Your code should then run successfully.

If you are facing any other issue, I would recommend first running the ‘Train Biped Robot to Walk Using Reinforcement Learning Agents’ shipped example on your end and contacting MathWorks support if you face any issue. The following is the link to the example:

https://www.mathworks.com/help/reinforcement-learning/ug/train-biped-robot-to-walk-using-reinforcement-learning-agents.html