Problems training deep neural network

I have a custom neural network with 3 hidden layers and I'm having trouble initializing it. The problem is that when training stops I get a very small gradient so no much learning in the last epochs but my error is still huge even for the training set! I've investigated the problem and it come out that my last layer of sigmas is saturated and there is no remarquable change for the weights of the early layers. I've red that this problem of vanishing gradient can be solved using the Relu transfer functions but now I have a huge error due to enormous weights. I'm stuck for 2 weeks tryings Relu and its variants (Leaky RElu and Elu ) with different training functions (trainrp and trainscg) with no success. I doubt that I need a good initialization but I can't find anything in Matlab other the NW and I thinks that's strange for Matlab. Can anyone help me or at least put me on the right truck ?! Thanks

Regards

Amine

[Greg Heath]

How are your inputs and targets normalized? i,e, what are the boundaries of each component?

Greg

Answer 1

[qi lu]

I encountered the same problem, is there a possible solution?