How to extract episode count when training reinforcement learning agents in Simulink?

I am currently conducting research on control using reinforcement learning with MATLAB and Simulink.

When training with the RL agent in Simulink, I would like to set different reward functions depending on the training episode (for example, designing a different reward function for the initial training phase and another for later phases).

I would like to ask if there is a way to access and utilize episode information in real time during training in Simulink.

Answer 1

[Angelo Yeo]

In the Simulink reinforcement learning environment, there is no explicit block to obtain the currently running episode. However, you can retrieve the episode number by using the trainingProgressMonitor function from the Deep Learning Toolbox and the logging features of rlDataLogger from the Reinforcement Learning Toolbox. For example, before training with the train function, you can create a logger object using trainingProgressMonitor and rlDataLogger, then set it up so that a callback function called myAgentEpisodeFinishedFcn is triggered every time an AgentEpisode ends. In this callback function, you can access the episodeCount value.

monitor = trainingProgressMonitor();
logger = rlDataLogger(monitor);
logger.EpisodeFinishedFcn = @myAgentEpisodeFinishedFcn;
trainingStats = train(agent,env,trainOpts,Evaluator=evl, Logger = logger);

The callback function is written as below.

function dataToLog = myAgentEpisodeFinishedFcn(data)
dataToLog = [];
persistent episodeCount
episodeCount = data.EpisodeCount;

% Here, a variable episodeNumber gets created in base workspace
assignin('base', 'episodeNumber', episodeCount); 
end

Afterwards, you can include the episodeNumber in the reward calculation part and modify the reward as you wish. You can find the detailed implementation of the above model in the attached file. (The original model is the Watertank example included in the official documentation.)