How to train network with additional epoch after end of iteration. As soon as I repeat training of neural network with GPU parallel computing, it stops with maximum mu.

I modified clolsed-loop NARX example, and try to use GPU parallel computing in training procedure.

At first, the maximum epoch was set to be 50,000, and training was conducted with the following command.

clonet = train(clonet,Xs,t,Xi,'useParallel','yes','useGPU','only');

After end of 50,000 epoch without any other termination (maximum mu or validation check), I think more training of neural netwrok is needed .

Thus, I typed the same command one more time in the command line (WIth that I hope I can train 50,000+50,000 = 100,000 epoch).

clonet = train(clonet,Xs,t,Xi,'useParallel','yes','useGPU','only');

However, as soon as I repeat training of neural network with GPU parallel computing, it stops with maximum mu (end at one epoch).

I think it is very starnge. In my opinion, if I set epoch to be 100,000 at first, this problem might not occur.

When I don't use GPU option during training (first and second), then repetition of training works well and results in better result than that with only first training.

I want to ask whether such repetition of network training doesn't works on GPU parallel computing option.

Is there any other method to train network with additional epoch after end of iteration ?

Thank you.

Answer 1

[Mahesh Taparia]

Hi

To restart the training process from the last epoch, first you need to load the trained weights/ checkpoints of last trained epoch. You can refer to this documentation link of training restart from the last epoch or else you can set the number of epochs as 100,000 as you mentioned.