How to determine the correct number of Validation Frequency during neural network training

Hello

How is the appropriate and optimal value for parameter Validation Frequency and mini batch size calculated?

Assuming we have 2,000 training and 200 validation data and 20 epochs

Is a value of 10 suitable for Validation Frequency and value of 20 for mini batch size?

Answer 1

[Ullah Nadeem]

Dear Lech!

Mini-batch size as good as large as your system memory can afford. Large mini-batch size results in more number of training samples to be learned by model which is good to generalize the problem and decrease the loss quickly.

VF: Validation Frequency

As far as I know, the selection of validation frequency depends on the following:

(1) The amount of training data

If the training data is large and the VF if small number (let's say after 30 iterations), the model will not learn enough while validating more oftenly results in long training time and maybe stop the training early if the 'ValidationPatience' is not set to 'Inf'.

(2) How fast the model learns the data well

If a problem is easy and the model learns it quickly, the VF supposed to be in range of 1/15th times of the iterations/epoch to 1/7th times of the iterations/epoch to let the network stop early and not gets overfit. But if the problem is complicated and model can't learns it quickly, then it is better to select the VF in range of 1/7th times of the iterations/epoch to 1/3th times of the iterations/epoch to let the model learns enough before validating.

Note: Iterations/epoch can be calculated as total number of training samples/mini-batch size.