how to normalize CNN-Data?

I got always NaN as output from my network and it might be possible that the network parameters diverge during training. Could somebody help me to fix this problem? This is my code:

X(:,:,3,60) = rand(500);

Y=randn(1,1,250000,60);

layers = [...

imageInputLayer([500 500 3])

convolution2dLayer(51,6)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

convolution2dLayer(20,9)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

convolution2dLayer(10,12)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

convolution2dLayer(6,12)

batchNormalizationLayer

reluLayer

maxPooling2dLayer(2,'Stride',2)

batchNormalizationLayer

convolution2dLayer(6,12)

reluLayer

maxPooling2dLayer(2,'Stride',2)

fullyConnectedLayer(250000)

regressionLayer;

];

options = trainingOptions('sgdm','InitialLearnRate',0.001, ...

'MiniBatchSize',miniBatchSize, ...

'MaxEpochs',15,'ExecutionEnvironment','cpu');

net = trainNetwork(X,Y,layers,options);

Answer 1

[Jayanti]

Since you are getting NaN values as output, it might be due to numerical instability during the training of a neural network.

You can make following changes to solve this issue:

1. Try to reduce the learning rate to 0.0001. A smaller learning rate will result in gradual updates to the network weights. This will help in stabilizing the training process by preventing any drastic changes to the network.
2. Implement gradient clipping to limit the magnitude of the gradients during backpropagation. This is crucial for preventing the exploding gradient problem, which can cause extremely large updates to the network weights.

You can refer to the following changes I have made to the code, and it is working fine now.

options = trainingOptions('sgdm','InitialLearnRate',0.0001, ...
    'MiniBatchSize',2, ...
    'MaxEpochs',15,'ExecutionEnvironment','cpu','GradientThreshold', 1);

Hope it helps!