rlDiscrete​Categorica​lActor not accepting a mix of rlNumericSpec and rlFiniteSetSpec objects - observation for a RL environment

I am looking for an example of which implements a mix of rlNumericSpec and rlFiniteSetSpec object in an RL environment (as mentioned here). Some of my observations are numerical/continuous whereas others are finite/discrete.

I created a set of obervations which is mixture of rlNumericSpec and rlFiniteSetSpec objects using the following code:

obsInfo_numeric = rlNumericSpec([4 1]);
obsInfo_finite = rlFiniteSetSpec([1 1]);
obsInfo = [obsInfo_numeric,obsInfo_finite];

and a set of actions using:

actInfo = rlFiniteSetSpec([1 2 3 4 5]);

I also created a network called 'actnet' with 4 inputs and 1 output:

But when I try to create an actor using the observations and actions, I am getting an error:

actor = rlDiscreteCategoricalActor(actnet,obsInfo,actInfo);

Answer 1

[Narvik]

Hi,

I understand that you faced an issue when using a combination of discrete('rlFiniteSetSpec') and continuous('rlNumericSpec') observation data specifications. The function 'rlDiscreteCategoricalActor' accepts a combination discrete and continuous observation data specifications. Please find an example in the documentation below :

https://in.mathworks.com/help/reinforcement-learning/ref/rl.function.rldiscretecategoricalactor.html#mw_b11b3d8e-9739-4cd2-8750-f0350de918f7

I advise you to check your neural network and action space and make sure that the input layers match the number of observation channels. Please find some helpful documentation links provided below :

https://in.mathworks.com/help/reinforcement-learning/ref/rl.util.rlnumericspec.html

https://in.mathworks.com/help/reinforcement-learning/ref/rl.util.rlfinitesetspec.html

https://in.mathworks.com/help/reinforcement-learning/ref/rl.function.rldiscretecategoricalactor.html

Hope this helps!

[Pruthwiraj Santhosh]

Hi Narvik,

Thank you for your response!

I believe the problem is with the mismatch in the calculation of the number of network input layers. But I can't seem to figure out how to calculate this for a heterogenous RL Dataspec object.

For example, If I create obsInfo and actInfo using the following commands

obsInfo_numeric = rlNumericSpec([4 1]);
obsInfo_finite = rlFiniteSetSpec([1 1]);
obsInfo = [obsInfo_numeric,obsInfo_finite];

actInfo = rlFiniteSetSpec([1 2 3 4 5]);

and then create a neural network using the following set of code

actnet = [featureInputLayer(x,"Name","obs") % x - number of input layers
    fullyConnectedLayer(20,"Name","fc1")
    reluLayer("Name","relu1")
    fullyConnectedLayer(20,"Name","fc2")
    reluLayer("Name","relu2")
    fullyConnectedLayer(20,"Name","fc3")
    reluLayer("Name","relu3")
    fullyConnectedLayer(5,"Name","act")
    softmaxLayer("Name","sfact")]

Can you tell me the value of 'x' in the above code for which the rlDiscreteCategoricalActor won't return an error?

actor = rlDiscreteCategoricalActor(actnet,obsInfo,actInfo,'ObservationInputNames',"obs")