Pattern recognition neural network : training process performance question

I am just starting using neural networks, but I am troubled by something when I use patternnet function. I have a set of about 200 data, each having 5 parameters (input is 5x200). I want the NN to classify those data into 7 classes, so my target data is a 7x200 matrix (eg. [0; 0; 1; 0; 0; 0; 0]). These data are quite "noisy".

net = patternnet(20);
net = train(net,inputs,targets);

My problem is that the results vary each time I restart the training process. What I mean is that if I delete this network and build another one with the same characteristics and the same inputs/targets, the performance of the NN vary greatly.

To give a better idea of my results, I can obtain 91% good calls (with the confusion matrix) one time, and the next time, it is 28%...

Is this normal, or is it a problem with the characteristics of the NN, or could It be my data?

Like I said, I just started using NN, and maybe it's the type of NN that is not correct for my situation, but I thought NN would be more steady in the training process.

Thanks.

[Kosai]

Each time a neural network is created, it has semi-random weights and bias values. Hence, when trained it will find a different solution and therefore have different input-output sensitivities.

To consistently get the same solution and input-output sensitivities, for the same network architecture trained on the same data, set the random seed generator to a fixed value (such as pi) before creating and training the network:

rand('seed',pi);

Answer 1

[Greg Heath]

Initial weights are chosen randomly. In order to duplicate previous runs you have to reset the RNG to the same initial state. I use a famous birthdate via rng(4151941). However, there are newer (better)ways to intialize the RNG, e.g. help/doc randstream.

As you have experienced, the ability to find a good local minimum is random. This is normal. My approach is to run Ntrials = 10 to 30 trials for every candidate value of H = numhidden (double for loop) and store the results in 3 matrices (trn/val/tst) with size(results) = [ Ntrials H] . My training goal is usually ~mean(var(t'))/100 (R^2 > ~0.99) where t is the training target with unit matrix columns, e.g., [ 0 0 0 1 0 0 ]'.

I then choose a net with the smallest trial value of H that achieves the minimum test set classification rate.

You may want to form an ensemble by combining several good designs. See design examples in the Newsgroup by searching on

heath newff Ntrials

Also try newfit, fitnet, patternnet & feedforwardnet.

Hope this helps.

Greg

[Julien]

Thanks! I will surely use this!

Answer 2

[Walter Roberson]

I do not have the background to know why, but I have seen enough postings here to know that NN training often uses the random number generator. To get the same network for the same input data, use rnd() or one of the other tools to set the state for the random number generator (be sure to set the right random generator, as there can be several simultaneous ones.)

[Julien]

Thanks for the answer! Although Greg's answer was a bit more complete... :)

[Greg Heath]

Walter: Why?

Answer: For a typical RW network node/connection-weight topology, an explicit solution of the nonlinear input-output relationship is untenable. Therefore, weight values that will mimimize the mean-square-error between outputs and design targets are found numerically via nonlinear multivariate optimization (NMO). The solution is made even more difficult by a large number of unknown weights. Multiple random initial trial solutions is a common approach in most NMO methods (e.g., gradient descent). The largest difficulty tends to be dealing with a plethora of non-optimal local minima in the solution space for unknown weights.

Hope this helps.

Greg