Why Training Set accuracy decrease dramatically after stopping the trainNetwork?

After stopping manually trainNetworktrainNetwork, the validation error dropped dramatically:

I tested the Training Set accuracy, and got also about 60%:

predY = classify(net,xTrain);

Any ideas what I'am doing wrong?

[Greg Heath]

Four curves and none of them labelled ??

[Sergy Stepura]

Legend in lower right corner:

[Don Mathis]

What is your network architecture? Does it contain dropoutLayers and later BatchNormlization layers?

[Sergy Stepura]

The network has simple architecture, 5 fully connected layers with batch normalization + Input layer + Output layer (softmax):

     1   ''   Image Input             120x1x4 images with 'zerocenter' normalization
     2   ''   Fully Connected         65 fully connected layer
     3   ''   Batch Normalization     Batch normalization
     4   ''   ReLU                    ReLU
     5   ''   Fully Connected         65 fully connected layer
     6   ''   Batch Normalization     Batch normalization
     7   ''   ReLU                    ReLU
     8   ''   Fully Connected         65 fully connected layer
     9   ''   Batch Normalization     Batch normalization
    10   ''   ReLU                    ReLU
    11   ''   Fully Connected         65 fully connected layer
    12   ''   Batch Normalization     Batch normalization
    13   ''   ReLU                    ReLU
    14   ''   Fully Connected         65 fully connected layer
    15   ''   Batch Normalization     Batch normalization
    16   ''   ReLU                    ReLU
    17   ''   Fully Connected         3 fully connected layer
    18   ''   Softmax                 softmax
    19   ''   Classification Output   crossentropyex

Answer 1

[Don Mathis]

Maybe your minibatch size is too small. The accuracy drop may be due to batchnormalization layers getting finalized, during which time the mean and variance of the incoming activations of each batchnorm layer are computed using the whole training set. If those full-batch statistics don't match the minibatch statistics very well, the finalized batchnorm layers will not be performing a very good normalization.

[Sergy Stepura]

I’m training on 30,000 data set and using batches of 500, do you sugest to use 30,000 batch size?

I’m want to see some stable performance before increasing to full 2,000,000 data set.

[Don Mathis]

You could try increasing the batch size iteratively to see whether that fixes the problem. I would try exponentially increasing: 1000, 2000, 4000, 8000, etc. Or you can just try the largest amount that will fit in your GPU memory right away.

[Don Mathis]

Also: Why does your plot show "Iterations per epoch: 1"? Were you using miniBatchSize=30000 in that run?

What are you passing to trainingOptions()?