Output Function to Save Net on Every Validation

I'm curious if it's possible to define an output function to spit out the current state of the network while training by using an output function to put that current net into a structure in the same way I have it defined to spit out [net,tr] = trainNetwork() when it finishes, but does so during training.

I can't use checkpoints because I am using an ADAM solver for my network.

1: Net,TR

2: Net, TR

3: Net, TR

4: Net, TR

etc.

[Ameer Hamza]

It seems that the outputFcn cannot save the network itself after each iteration. Is saving just the state of network training enough?

Answer 1

[Ameer Hamza]

If you just want to save the training states, then try the following example. It is adapted from this example: https://www.mathworks.com/help/releases/R2020a/deeplearning/ref/trainingoptions.html#bvniuj4

[XTrain,YTrain] = digitTrain4DArrayData;

idx = randperm(size(XTrain,4),1000);
XValidation = XTrain(:,:,:,idx);
XTrain(:,:,:,idx) = [];
YValidation = YTrain(idx);
YTrain(idx) = [];

layers = [
    imageInputLayer([28 28 1])
    
    convolution2dLayer(3,8,'Padding','same')
    batchNormalizationLayer
    reluLayer   
    
    maxPooling2dLayer(2,'Stride',2)
    
    convolution2dLayer(3,16,'Padding','same')
    batchNormalizationLayer
    reluLayer   
    
    maxPooling2dLayer(2,'Stride',2)
    
    convolution2dLayer(3,32,'Padding','same')
    batchNormalizationLayer
    reluLayer   
    
    fullyConnectedLayer(10)
    softmaxLayer
    classificationLayer];

options = trainingOptions('sgdm', ...
    'MaxEpochs',8, ...
    'ValidationData',{XValidation,YValidation}, ...
    'ValidationFrequency',30, ...
    'Verbose',false, ...
    'Plots','training-progress', ...
    'OutputFcn', @outFcn);

global training_state
training_state = [];
net = trainNetwork(XTrain,YTrain,layers,options);

function stop = outFcn(info)
    global training_state
    training_state = [training_state info];
    stop = false;
end

Use of the global variable can be avoided if you define your own handle class and pass it to the outFcn. However, if you are fine with the use of global, then it shouldn't be an issue.

[Grant Anderson]

Hey Ameer. I'm running into issues with the Global variable needing to be defined by the outer function.

I've tried multiple times to work around this but can't get it to work.

This may be because I have my trainNetwork inside of a function that I call (so I can feed it different variables for network configuration). I've tried putting the global training_state in different parts of each function but have no luck.

Will the training_state as it is defined even be set up to re-build the network I choose to? or is the training_state only capturing the characters such as 'final' (i.e. the phase of training the network is in)

[Ameer Hamza]

The information captured by training_state cannot be used to rebuild the network. It only gives info like training accuracy, validation accuracy, etc. I don't think MATLAB provides a way to capture the entire network state during training.

Also, the global variable only needs to be defined in the base workspace and the workspace of the function it is being used, i.e., outFcn. It does not need to be defined anywhere else. Can you just show the general structure of your code, and where are you trying to define the global variables?

[Grant Anderson]

This is the function in which I feed in some parameters to train a neural network. It allows me to re-size the fully-connected-layers and neurons.

function [net,tr] = betNet(X,y,X_test,y_test,X_cv,y_cv,maxE,NHL,fcls)



%% ===== Setting up DNN =====
%Sets up our FCL
fcl1 = fullyConnectedLayer(fcls,'BiasInitializer','narrow-normal');
fcl2 = fullyConnectedLayer(2,'BiasInitializer','ones');

ip = sequenceInputLayer(size(X,1),'Normalization','zerocenter');
sml = softmaxLayer('Name','sml');
options = trainingOptions('adam',...
    'MaxEpochs',maxE,...
    'ExecutionEnvironment','gpu',...
    'Shuffle','every-epoch',...
    'MiniBatchSize',64,...
    'ValidationFrequency',50,...
    'ValidationData',{X_cv,y_cv},...
    'OutputFcn', @outFcn)


layers = [ip repmat(fcl1,1,NHL) fcl2 softmaxLayer classificationLayer];
% 
global training_state
training_state = [];


%% ===== Training NN =====
[net,tr] = trainNetwork(X,y,layers,options);

function stop = outFcn(info)
     global training_state
     training_state = [training_state info];
     stop = false;
end

end

[Ameer Hamza]

If you want to check the value of training_state in the base workspace after the execution of your function, then you should also run the following line in the command window before calling your function.

global training_state