Validation Accuracy on Neural network

Hello..I wonder if any of you who have used deep learning on matlab can help me to troubleshoot my problem. I don't understand why I got a sudden drop of my validation accuracy at the end of the graph? It's a simple network with one convolution layer to classify cases with low or high risk of having breast cancer. After the final iteration it displays a validation accuracy of above 80% but then suddenly it dropped to 73% without an iteration. I don't understand that.

Here's my code

%set training dataset folder
digitDatasetPath = fullfile('C:\Users\UOS\Documents\Desiree Data\Run 
2\dataBreast\training2');

%training set
imdsTrain = imageDatastore(digitDatasetPath, ...
'IncludeSubfolders',true,'LabelSource','foldernames');

%set validation dataset folder
validationPath = fullfile('C:\Users\UOS\Documents\Desiree Data\Run 
2\dataBreast\validation2');

%testing set
imdsValidation = imageDatastore(validationPath, ...
'IncludeSubfolders',true,'LabelSource','foldernames');

%create a clipped ReLu layer
layer = clippedReluLayer(10,'Name','clip1');

% define network architecture
layers = [
imageInputLayer([256 256 1]);

% conv_1
convolution2dLayer(3,32,'Stride',1)
batchNormalizationLayer
clippedReluLayer(10);
maxPooling2dLayer(2,'Stride',2)

%fc
fullyConnectedLayer(100)
dropoutLayer(0.7,'Name','drop1');

%fc
fullyConnectedLayer(25)
dropoutLayer(0.8,'Name','drop2'); 

% fc layer
fullyConnectedLayer(2)
softmaxLayer
classificationLayer];

% specify training option
options = trainingOptions('adam', ...
'InitialLearnRate',0.001, ...
'MaxEpochs',15, ...
'Shuffle','every-epoch', ...
'ValidationData',imdsValidation, ...
'ValidationFrequency',30, ...
'Verbose',false, ...
'Plots','training-progress');

% train network using training data
net = trainNetwork(imdsTrain,layers,options);

% classify validation images and compute accuracy
YPred = classify(net,imdsValidation);
YValidation = imdsValidation.Labels;

%calculate accuracy
accuracy = sum(YPred == YValidation)/numel(YValidation);

[Don Mathis]

Does it still happen if you use a much larger MiniBatchSize, say 1000 or 2000?

[Andrik Rampun]

Thanks for your reply Don. Do you mean using something like convolution2dLayer(3,1000,'Stride',1)

My computer can't actually support (memory issue) this is this is what you meant.

[Jon Cherrie]

I think Don means the MiniBatchSize in the training options,

• https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html#bu59f0q-MiniBatchSize

The drop in validation accuracy in the plot might be the same as this:

• https://www.mathworks.com/matlabcentral/answers/445800-why-do-i-see-a-drop-or-jump-in-my-final-validation-accuracy-when-training-a-deep-learning-network

[Andrik Rampun]

Thanks Jon. The second link is interesting. I will try to do that and get back with some results. But in that case does it mean I can't use batch normalization anymore? My previous experience using caffe+DIGITS using batch normalization gave me better results in most of the cases.

[Andrik Rampun]

Hi Jon, Don,

Here I train the network without batch normalization and I got no jump/drop at the end of the training but as you can see the results are are really low. I notice in matlab tutorial they also using batch normalization and when I run the code I didn't get a jump/drop at the end of the iteration.

% set training dataset folder
digitDatasetPath = fullfile('C:\Users\UOS\Documents\Desiree Data\Run 2\dataBreast\training2');

imdsTrain = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');


% set validation dataset folder
validationPath = fullfile('C:\Users\UOS\Documents\Desiree Data\Run 2\dataBreast\validation2');

imdsValidation = imageDatastore(validationPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');

% create a clipped ReLu layer
layer = clippedReluLayer(10,'Name','clip1');

% define network architecture
layers = [
    imageInputLayer([256 256 1], 'Normalization', 'none')
    
    % conv_1
    convolution2dLayer(3,16,'Stride',1)
    %batchNormalizationLayer
    clippedReluLayer(10);
    maxPooling2dLayer(2,'Stride',2)
    
    
    % fc layer
    fullyConnectedLayer(2)
    softmaxLayer
    classificationLayer];

% specify training option
options = trainingOptions('adam', ...
    'InitialLearnRate',0.001, ...
    'MaxEpochs',15, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',30, ...
    'Verbose',false, ...
    'MiniBatchSize',32, ...
    'Plots','training-progress');

% train network using training data
net = trainNetwork(imdsTrain,layers,options);

% classify validation images and compute accuracy
YPred = classify(net,imdsValidation);
YValidation = imdsValidation.Labels;

accuracy = sum(YPred == YValidation)/numel(YValidation)

[amir aslam]

I have done the same experiment but accuracy does not drop

code:

clc;

clear;

%set training dataset folder

digitDatasetPath = fullfile('C:\Program Files\MATLAB\R2018aaa\toolbox\nnet\nndemos\nndatasets\Experiments\Train');

%training set

imdsTrain = imageDatastore(digitDatasetPath, ...

'IncludeSubfolders',true,'LabelSource','foldernames');

%set validation dataset folder

validationPath = fullfile('C:\Program Files\MATLAB\R2018aaa\toolbox\nnet\nndemos\nndatasets\Experiments\Test');

%testing set

imdsValidation = imageDatastore(validationPath, ...

'IncludeSubfolders',true,'LabelSource','foldernames');

%create a clipped ReLu layer

%layer = clippedReluLayer(10,'Name','clip1');

% define network architecture

layers = [

imageInputLayer([48 48 3]);

% conv_1

convolution2dLayer(3,32,'Stride',1)

batchNormalizationLayer

reluLayer;

maxPooling2dLayer(2,'Stride',2)

%fc

fullyConnectedLayer(100)

dropoutLayer(0.7,'Name','drop1');

%fc

fullyConnectedLayer(25)

dropoutLayer(0.8,'Name','drop2');

% fc layer

fullyConnectedLayer(7)

softmaxLayer

classificationLayer];

% specify training option

options = trainingOptions('adam', ...

'InitialLearnRate',0.0001, ...

'MaxEpochs',15, ...

'Shuffle','every-epoch', ...

'ValidationData',imdsValidation, ...

'ValidationFrequency',30, ...

'Verbose',false, ...

'Plots','training-progress');

% train network using training data

net = trainNetwork(imdsTrain,layers,options);

% classify validation images and compute accuracy

YPred = classify(net,imdsValidation);

YValidation = imdsValidation.Labels;

%calculate accuracy

accuracy = sum(YPred == YValidation)/numel(YValidation);

[Sridharan K]

i got 100% accuracy. thanks for this program.

[Santhosh Surya Kiran]

What u got is wrong..

Answer 1

[Andrik Rampun]

Some updates. I got similar results (sudden drop) at the end of the graph. I find this really strange.

% set training dataset folder
digitDatasetPath = fullfile('C:\Users\UOS\Documents\Desiree Data\Run 2\dataBreast\training2');

imdsTrain = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');


% set validation dataset folder
validationPath = fullfile('C:\Users\UOS\Documents\Desiree Data\Run 2\dataBreast\validation2');

imdsValidation = imageDatastore(validationPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');

% create a clipped ReLu layer
layer = clippedReluLayer(10,'Name','clip1');

% define network architecture
layers = [
    imageInputLayer([256 256 1], 'Normalization', 'none')
    
    % conv_1
    convolution2dLayer(3,16,'Stride',1)
    batchNormalizationLayer
    clippedReluLayer(10);
    maxPooling2dLayer(2,'Stride',2)
    
    % conv_2
    convolution2dLayer(3,16,'Stride',1)
    batchNormalizationLayer
    clippedReluLayer(10);
    maxPooling2dLayer(2,'Stride',2)
   
    % conv_3
    convolution2dLayer(3,32,'Stride',1)
    batchNormalizationLayer
    clippedReluLayer(10);
    maxPooling2dLayer(2,'Stride',2)
    
    % conv_4
    convolution2dLayer(3,64,'Stride',1)
    batchNormalizationLayer
    clippedReluLayer(10);
    maxPooling2dLayer(2,'Stride',2)
    
    % conv_5
    convolution2dLayer(3,128,'Stride',1)
    batchNormalizationLayer
    clippedReluLayer(10);
    maxPooling2dLayer(2,'Stride',2)
    
    % conv_6
    convolution2dLayer(3,256,'Stride',1)
    batchNormalizationLayer
    clippedReluLayer(10);
    maxPooling2dLayer(2,'Stride',2)
    
    % fc5
    fullyConnectedLayer(500)
    dropoutLayer(0.5,'Name','drop1'); 
    
        
    % fc5
    fullyConnectedLayer(250)
    dropoutLayer(0.5,'Name','drop2');
    
    
    % fc5
    fullyConnectedLayer(50)
    dropoutLayer(0.5,'Name','drop3'); 
    
    % fc layer
    fullyConnectedLayer(2)
    softmaxLayer
    classificationLayer];

% soecify training option
options = trainingOptions('adam', ...
    'InitialLearnRate',0.001, ...
    'MaxEpochs',30, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',30, ...
    'Verbose',false, ...
    'Plots','training-progress');

% train network using training data
net = trainNetwork(imdsTrain,layers,options);

% classify validation images and compute accuracy
YPred = classify(net,imdsValidation);
YValidation = imdsValidation.Labels;

accuracy = sum(YPred == YValidation)/numel(YValidation)

%YPred = predict(net,X);

analyzeNetwork(net)

[Don Mathis]

What is the size of your training set?

[Andrik Rampun]

Thanks for your reply again Don. The training set (368- low risk, 372-high risk), for validation set (59- low risk, 40-high risk)

[Andrik Rampun]

Hi Don,

I used a very large minibatchsize but still got the same results (a sudden drop/jump)

% specify training option
options = trainingOptions('adam', ...
    'InitialLearnRate',0.001, ...
    'MaxEpochs',15, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',30, ...
    'Verbose',false, ...
    'MiniBatchSize',2048, ...
    'Plots','training-progress');

[Don Mathis]

Your training set has 740 observations, which is a very small training set size, especially for such a large network. Now that you have specified MiniBatchSize=2048, each minibatch contains the entire training set, which we can verify by noticing that the number of iterations is equal to the number of epochs.

Looking at your final plots, I would say there's a good chance that this fixed the problem. The finalized loss is visually identical to the unfinalized loss, and the finalized error rate jumps about 5%, which is probably 5 of your 99 validation observations. I'm not sure why it jumps at all, but 5 observations switching their class seems like noise to me.

You need to run many more iterations. 15 is not enough. I suggest running for at least 100, or until validation error levels off.

Oh, and if you can, try to get your hands on 1 million observations :-)

[Andrik Rampun]

Thanks Don. I'm training it now with 2048 mini batch and 100 epochs. I will get back as soon as I have the results.

[Andrik Rampun]

Hi Don,

I have finished the training and got higher jump/drop. As you can see the drop is over 10% now.

I find this really strange as I don't have this problem on Caffe+DIGITS. If batchnormalization causes the problem then what's the solution?

I have also increased the number of training+validation and testing.

Training (low risk=896, high risk=712)

Validation (low risk=59, high risk=67)

Testing (low risk =52, high risk=50)

options = trainingOptions('adam', ...
    'InitialLearnRate',0.001, ...
    'MaxEpochs',200, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'MiniBatchSize',2048, ...
    'ValidationFrequency',30, ...
    'Verbose',false, ...
    'Plots','training-progress');

[Don Mathis]

What version of MATLAB are you using? (Which of R2017b, R2018a, R2018b?)

[Andrik Rampun]

I'm using Matlab 2018b.

[Don Mathis]

There are 2 ways I'm aware of that BatchNormalization layers can cause this problem:

(1) There is some layer or data transformation that occurs only during training (not during prediction), and which precedes a batchnorm layer (by any distance). For example, if a dropout layer precedes a batchnorm layer, or if your training images have noise added, and your validation images do not.

(2) The mean and variance of your minibatch activations don't match the mean and variance of the full dataset activations.

Unless you're adding noise to your training images in some way, it doesn't appear that either of these scenarios apply to you.

[Andrik Rampun]

Do you have any suggestions how can I solve this problem? Or pahaps this doesn't really matter as long as I can get good results in my testing set?

What I don't want is doing wrong things in my experiemnt implementation

[Don Mathis]

It doesn't look like you're doing anything wrong. I think this is just how these layers behave under these circumstances.

But it's difficult to draw precise conclusions from the experiments you've done because you have used 3 very different networks. The first had 1 conv, 3 fullyConnected, and very high dropout. The second had 1 conv, 1 fullyConnected, and no dropout. The third had 6 conv, 4 fullyConnected, and moderate dropout.

My next hypothesis would be that batchnorm may show this 'drop' effect when the network is too large for the dataset size.

My best guess for removing the 'drop' issue would be:

1. Continue to use a large minibatch size (e.g., 2000 or the size of the training set).
2. Start with small networks (like your first one) and work up to larger ones gradually.
3. Use a dropout rate <= 0.5.
4. Do not put dropout before batchnorm in the network.

[Andrik Rampun]

Thanks Don,

I will try your suggestions. Really appreaciate your help.

[Andrik Rampun]

Hi Don,

When you said 'Do not put dropout before batchnorm in the network.'

I think for all my networks I didn't put dropout in the conv layer right? because all my dropout layers are are in the FC layers. Is that right?

[Don Mathis]

Your networks were ok in that regard. You only put dropout after batchnorm. So that's not why you're getting the 'drop' effect.

[Andrik Rampun]

Hi Don,

Just to follow up some updates here...

I ran the following code

clear all
% set training dataset folder
digitDatasetPath = fullfile('/data/md1ybr/dataBreast/Run1/training');

imdsTrain = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');


% set validation dataset folder
validationPath = fullfile('/data/md1ybr/dataBreast/Run1/validation');

imdsValidation = imageDatastore(validationPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');

% create a clipped ReLu layer
layer = clippedReluLayer(5,'Name','clip1');

% define network architecture
layers = [
    imageInputLayer([256 256 1], 'Normalization', 'zerocenter')
    
    % conv_1
    convolution2dLayer(7,64,'Stride',1)
    batchNormalizationLayer
    clippedReluLayer(5);
    maxPooling2dLayer(2,'Stride',2)

    % fc2
    fullyConnectedLayer(3136)
    dropoutLayer(0.1,'Name','drop1');     
        
    % fc3
    fullyConnectedLayer(3136)
    dropoutLayer(0.1,'Name','drop2');

     
    % fc layer
    fullyConnectedLayer(2)
    softmaxLayer
    classificationLayer];

% soecify training option
options = trainingOptions('adam', ...
    'InitialLearnRate',0.001, ...
    'MaxEpochs',200, ...
    'Shuffle','every-epoch', ...
    'ValidationData',imdsValidation, ...
    'MiniBatchSize',2048, ...
    'ValidationFrequency',30, ...
    'Verbose',true);

% train network using training data
net = trainNetwork(imdsTrain,layers,options);

% classify validation images and compute accuracy
YPred = classify(net,imdsValidation);
YValidation = imdsValidation.Labels;

accuracy = sum(YPred == YValidation)/numel(YValidation);
disp('Training Acc:');
disp(accuracy);



% set validation dataset folder
testingPath = fullfile('/data/md1ybr/dataBreast/Run1/testing');


imdsTesting = imageDatastore(testingPath, ...
    'IncludeSubfolders',true,'LabelSource','foldernames');

ZValidation = imdsTesting.Labels;
ZPred = predict(net,imdsTesting);

pre=[];
for i=1:size(ZPred,1)
   if ZPred(i,1)>ZPred(i,2)
       pre=[pre;1];
   else
       pre=[pre;2];
   end
end

testAccuracy = sum(pre == grp2idx(ZValidation))/numel(ZValidation);
disp('Testing Acc:');
disp(testAccuracy);

I got the following error (I don't think the size of the image is that large to be honest.)

Error using trainNetwork (line 150)
Maximum variable size allowed on the device is exceeded.

Error in brestDensityClassification5v4a (line 53)
net = trainNetwork(imdsTrain,layers,options);

Caused by:
    Error using gpuArray
    Maximum variable size allowed on the device is exceeded.

[Don Mathis]

3136 hidden units in each FC layer times batchsize of 2048 adds up to a lot of memory. It's trying to put the whole batch on the GPU at once and failing. This is a common problem. You just need to shrink either the batchsize or parts of the network until it fits.

[Andrik Rampun]

Hi Don,

When you said shirink batchsize do you mean my MinibatchSize?

Another question is I can even reduce my hidden units in my FC layer?

Thanks

[Don Mathis]

Yes, MiniBatchSize. And I meant the outputSize of your fullyConnectedLayers could be reduced to something smaller than 3136:

fullyConnectedLayer(3136)

Answer 2

[Saira]

Hi,

I have 5600 training images. I have extracted features using Principal Component Analysis (PCA). Then I am applying CNN on extracted features. My training accuracy is 30%. How to increase training accuracy?

Feature column vector size: 640*1

My training code:

% Convolutional neural network architecture

layers = [

imageInputLayer([1 640 1]);

reluLayer

fullyConnectedLayer(7);

softmaxLayer();

classificationLayer()];

options = trainingOptions('sgdm', 'Momentum',0.95, 'InitialLearnRate',0.0001, 'L2Regularization', 1e-4, 'MaxEpochs',5000, 'MiniBatchSize',8192, 'Verbose', true);

Answer 3

[Sevda Kemba]

@Andrik Rampun Hello. In Matlab, we load the data set with code and limit it in deep learning. But when we train, validation accuracy stays between 40-50%. What can we do to increase it to 90%? We would be very happy if you could help.

Answer 4

[Sevda Kemba]

@Saira Hello. In Matlab, we load the data set with code and limit it in deep learning. But when we train, validation accuracy stays between 40-50%. What can we do to increase it to 90%? We would be very happy if you could help.