Neural Networks - Break down of training, testing and validation groups

Good morning,

I am trying to run a NN, for a regression type problem, with a different number of neurons each time. Then I have a measure of how each NN has performed. However, I would like to break this down into how each of the three groups (training, testing and validation) performed, but I can't get the index values of each group. Here is an excerpt from my code:

    trainFcn = 'trainbr';
    net = feedforwardnet;
    % Setup Division of Data for Training, Validation, Testing
    net.divideParam.valRatio = 15/100;
    net.divideParam.testRatio = 35/100;
    net.divideParam.trainRatio = 50/100;
    net.trainFcn = trainFcn;

    for neurons = 5:5:100
        net.layers{1}.size = neurons;
        net = configure(net,PWL',target');
        [net,tr] = train(net,PWL',target');
    end

I understood that I should be able to find the index' of the three groups using:

tr.trainInd;
tr.testInd;
tr.valInd;

However, my tr.valInd is ALWAYS empty and the ratios of tr.trainInd and tr.testInd are never as set. Can someone please tell me what I am doing wrong? Even when I leave the division of the three groups to default I don't get anything in tr.valInd. I've looked through the manual and in their example they have index values in each of the three groups, but they don't show the whole example so I can't replicate.

Please help me,

Thanks

ps. Another idea is to use divideind instead of dividerand, but I can't work out how and when to enter the index'. Help with this would also be appreciated.

[jlt199]

This is what I get output for tr. You can see that the division of training, test and validation data isn't as I wanted. Any ideas?

>> tr

tr = 

struct with fields:

        trainFcn: 'trainbr'
      trainParam: [1×1 struct]
      performFcn: 'mse'
    performParam: [1×1 struct]
        derivFcn: 'defaultderiv'
       divideFcn: 'divideind'
      divideMode: 'sample'
     divideParam: [1×1 struct]
        trainInd: [1×7846 double]
          valInd: []
         testInd: [1×1385 double]
            stop: 'Maximum epoch reached.'
      num_epochs: 1000
       trainMask: {[1×9231 double]}
         valMask: {[1×9231 double]}
        testMask: {[1×9231 double]}
      best_epoch: 1000
            goal: 0
          states: {1×10 cell}
           epoch: [1×1001 double]
            time: [1×1001 double]
            perf: [1×1001 double]
           vperf: [1×1001 double]
           tperf: [1×1001 double]
              mu: [1×1001 double]
        gradient: [1×1001 double]
            gamk: [1×1001 double]
             ssX: [1×1001 double]
        val_fail: [1×1001 double]
       best_perf: 39.3436
      best_vperf: NaN
      best_tperf: 1.2468e+03

[Greg Heath]

I have published more than one examples of this in the NEWSGROUP. I will post at least one URL.

http://www.mathworks.com/matlabcentral/newsreader/search_results.html?dur=all&page=1&search_string=best_perf&sort=date_up

Hope this helps.

Greg

Answer 1

[Greg Heath]

The bottom line is that TRAINBR has at least one bug w.r.t. datadivision.

 trainbr bug

 http://www.mathworks.com/matlabcentral/newsreader/search_results.html?dur=all&page=1&search_string=trainbr+bug&sort=date_up

You probably will get more consistent results if you use TRAINLM with MSEREG

Hope this helps.

Thank you for formally accepting my answer

Greg

Answer 2

[Benhur Aysin]

To enable validation performance when using trainbr as the training function you must set max_fail to a positive number. 6 is usually the number for many training functions. max_fail=0 disables validation performance.