Performance estimate of pattern recognition tool

HI. I am creating a neural network using nprtool. I have generated the code and got the results in confusion matrix. In the end I get a perfoemace variable. I am unable to understand that what should be the value of this variable, I mean the range. Can any one please tell me that? I am appending my lines of code and my result.

% Test the Network outputs = net(inputs); errors = gsubtract(targets,outputs); performance = perform(net,targets,outputs)

My output is

• performance =

    0.4772*

My 2nd question is that how can i present the network generated by nprtool to new values? I don't understand the concept.

[Nida]

HI. I am creating a neural network using nprtool. I have generated the code and got the results in confusion matrix. In the end I get a perfoemace variable. I am unable to understand that what should be the value of this variable, I mean the range. Can any one please tell me that? I am appending my lines of code and my result.

% Test the Network outputs = net(inputs); errors = gsubtract(targets,outputs); performance = perform(net,targets,outputs)

My output is

performance = 0.4772* My 2nd question is that how can i present the network generated by nprtool to new values? I don't understand the concept.

Answer 1

[Greg Heath]

If you are using patternnet and/or nprtool the most important results are the error rates for the different classes. These are displayed in the confusion matrix. Each case is different because the number of classes may be different and the importance of each class may be different. Unfortunately, error rates are not continuous. Therefore it is very difficult to find an acceptable objective function based on error rates to be used in a numerical optimization routine.

The standard alternative approach is to minimize MSEtrn using gradient descent and to hope that the resulting the class distribution of trn/val/tst error rates is acceptable. If not, tradeoffs can be implemented in a number of ways. One way is to use the error weight function EW, as an input to TRAIN. Another is to add noisy duplicates of small classes with high error rates because their influence on the objective function was too small.

The output perf is the overall MSE. This includes all classes as well as training, validation and testing data. The breakdown of both trn/val/tst errorrate and MSE for all classes is readily achievable using the command line approach and training record tr obtained as an output of TRAIN.

I am not sure how or if both breakdowns are available using the GUI. The confusion matrix obviously has the class error rate breakdown. Not sure about the rest.

I just ran nprtool. It give the trn/val/tst break down of MSE and %E.

The confusion matrices give the trn/val/tst breakdown of class %E.

However, I didn't see a trn/val/tst breakdown of class MSE.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Nida]

thankyou so much for ur ans sir. I have seen the results in tr. I am having a simple 2 class problem. The output value is found to be as low as 0.0016 using 'trainscg'. With 'trainlm' however, the value is '3.6129e-05'. Is this fine? which of them is better? I am talking about the overall MSE here.

The second thing is that i dn understand that how much accurate values should be in confusion matrix? is it fine to get 99.4% answer? or the answer should be from the range, 90-95%? I am talking about True positives and true negatives correct results.

[Greg Heath]

I generally use MSEgoal = 0.01*Ndof*MSE00a/Ntrneq

Ntrneq = prod(size(target(:,trainInd)))% = Ntrn*O Number of training equqtions

Nw = (I+1)*H+(H+1)*O %Number of unknown weights for an I-H-O MLPNN

Ndof = Ntrneq - Nw % Estimation degrees of freedom (see Wikipedia)

MSE00a = mean(var(target'))

This yields an adjusted Rsquare >= 0.99 (see Wikipedia)

The goal for error rate depends on you and your particular problem. However, there is no known relationship between PctErr and MSE.

[Nida]

Thank you so much greg. I have done it all using some other research as my work had to be finished before July. Your reply will help me in further exploration and results for future

[afef]

@Nida please i need the code can you please send it to me i'm using the same thing like you and i need it for my master project that i have to finish it before july .this is my e-mail : [e-mail] Thanks

[Walter Roberson]

It is unlikely that Nida is still reading here, 4 years later. Nida will not be notified about your comment. You could try clicking on Nida's name above to reach their profile and then use the Contact button to send Nida email.