How to build an Actor-Critic model with shared layers?

Hi.

I'm trying to build an Actor-Critic model uisng Reinforcement Learning Toolbox.

What I'm currently intending is to share low level features over Actor and Critic, like this.

-----------------------------

inputImage

|

backboneNet

| |

actorNet criticNet

-------------------------------

So, I created three LayerGraphs for the backbone, as well as actor and critic, and connected them.

And then, the next steps are the same as the guideline of generating AC agent, like this.

%% Backbone Network
% ...(Some codes to define the backbone layers)...
backboneNet = createLgraphUsingConnections(all_layers, all_connections) ;

%% Critic Network
% ...(Some codes to define the critic layers)...
criticNet = layerGraph(fullyConnectedLayer(1, 'Name', 'critic_FC')) ;

all_layers = [...
    backbone.Layers
    criticNet.Layers] ;
all_connections = [...
    backbone.Connections
    criticNet.Connections] ;

connection = subgraphConnections(backbone.Layers(end), criticNet.Layers(1));
all_connections = [all_connections; connection];

criticNet = createLgraphUsingConnections(all_layers, all_connections) ;
critic = rlValueRepresentation(...
    criticNet, obsInfo,...
    'Observation', {'inputImage'},...
    criticOpts);

%% Actor Network
% ...(Some codes to define the actor layers)...
actorNet_concatOut = layerGraph(concatenationLayer(1, 2, 'Name', 'mean&sdev'));

all_layers = [...
    backbone.Layers
    actorNet_meanPath.Layers
    actorNet_sdevPath.Layers
    actorNet_concatOut.Layers] ;

all_connections = [...
    backbone.Connections
    actorNet_meanPath.Connections
    actorNet_sdevPath.Connections
    actorNet_concatOut.Connections] ;

connection1 = subgraphConnections(...
    backbone.Layers(end),...
    [actorNet_meanPath.Layers(1) actorNet_sdevPath.Layers(1)]);
connection2 = subgraphConnections(...
    [actorNet_meanPath.Layers(end) actorNet_sdevPath.Layers(end)],...
    actorNet_concatOut.Layers(1));
all_connections = [all_connections; connection1; connection2];

actorNet = createLgraphUsingConnections(all_layers, all_connections) ;
actor = rlStochasticActorRepresentation(...
    actorNet, obsInfo, actInfo,...
    'Observation', {'inputImage'},...
    actorOpts);

%% Agent
agent = rlACAgent(actor,critic,agentOpts);

But If I put in the backboneNet as above, the actorNet gradients appear as zero so the actorNet doen't learn anything.

When I was not using the backboneNet, everything was fine.

I guess I'm doing something wrong or misunderstand how to define shared layers.

Is there anybody know how to define shared layers over the actor and critic?

Thanks

[Bradley Fourie]

Hi Heesu,

I am attempting something similar, have you perhaps figured out what the problem was?

Kind regards,

Brad