Error with using loggedSignals in the reset function while creatin RL environment using custom functions

Hi,

I'm trying to train an rlPGAgent using a dataset I have (in a cell array variable called blocks in the following code). As a result, I'm defining a custom reset function that after each episode would take the next row from my dataset as the new initial observation. But I can't get the loggedSignal property working properly. Maybe I'm understanding it wrong, but shouldn't it be initialize when the rlFunctionEnv is called?

Here's my custom reset function:

function [initialObservation, LoggedSignals] = myResetFunction(blocks)
    % Start a new trial
    LoggedSignals.currentTrial = LoggedSignals.currentTrial + 1;
    if LoggedSignals.currentTrial > numel(blocks)
        LoggedSignals.currentTrial = 1; % Loop back to the first trial if it's gone through all of them
    end
    % Reset timestep for new trial
    LoggedSignals.currentTimestep = 1;

    % Set the initial state for the new trial
    initialObservation = [blocks{LoggedSignals.currentTrial,2}(1,2),... % Initial position
                          0,... % Initial hold (always 0)
                          0,... % Initial speed (always 0)
                          blocks{LoggedSignals.currentTrial,5}]; % Required hold time for this trial
    LoggedSignals.currentState = initialObservation;
end

And I call the function as a handle as follows:

ResetHandle = @() myResetFunction(blocks);
StepHandle = @(action,LoggedSignals) myStepFunction(action,LoggedSignals,blocks);

observationInfo = rlNumericSpec([1 4], 'LowerLimit', [0, 0, -30, 0], 'UpperLimit', [31, 5, 30, 1]);
actionInfo = rlNumericSpec([1 1], 'LowerLimit', -30, 'UpperLimit', 30);

env = rlFunctionEnv(observationInfo, actionInfo, StepHandle,ResetHandle);

The myResetFunction function alone (or when running the rlFunctionEnv function) runs into an error:

Unrecognized function or variable 'LoggedSignals'.

I'm using MATLAB R2021b, but I also tried Info (which seems to be replacing LoggedSignals in 2023b), but same issue. The issue is resolved if I initialized 'LoggedSignals' outside the function, and passed it to the reset function as an input argument, but that would cause the training to get stuck on the first trial, as the variable wouldn't update between reset and step function calls during the training process! Also declaring it as global causes a lot of other issues!

Does anyone know why this is happening and how I can potentially resolve it? Thanks for your help in advance!

Answer 1

[Emmanouil Tzorakoleftherakis]

The very first line in our reset function is

LoggedSignals.currentTrial = LoggedSignals.currentTrial + 1;

How would the function know what LoggedSignals.currentTrial value is? Hence the error. Either pass is as an argument or make it persistent and initialize it, or make it a class variable if you create the RL environment through the class template.

LoggedSignals is not a stored/shared variable in the RL environment. You can use it like that but it's up to you to make sure it's visible where it needs to be

[Daniel Ramandi]

Defining a class for the environment and making it a class variable worked. Thanks.