How to reproduce exact results of a patternnet network using feedforwardnet network.

I am trying to reproduce the results of a neural network created by "patternnet" function, using the "feedforwardnet" function instead. I fix

• the architecture
• the initialization method, and
• the seed in the randStream (for generating random numbers)

to be the same. However I still can't get identical results. Although I use the same initialization function I found that the difference is in the initial weights and biases. why are the weights and biases initialized differently? Debugging the functions I found that the "init" function is called more times than expected using the "patternnet" function. Why is this?

I post some code and results to prove that random number generator is not an issue. (A similar question has been asked here, but the problem has not been resolved).

TrainFeatNN = SelectedFeat40.Train;
TrainObsNN = trainToReduce.ObsClasses(:,2:3);
testInd = false(size(SelectedFeat40.Train,1),1);
testInd2 = false(size(SelectedFeat40.Train,1),1);
testInd3 = false(size(SelectedFeat40.Train,1),1);
testInd4 = false(size(SelectedFeat40.Train,1),1);
posclass = 1;
x = TrainFeatNN';
t = TrainObsNN'; 
%
%
%%Neural Networks patternnet #2
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize, trainFcn);
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','fixunknowns'};
net.divideFcn = 'divideblock';  
net.divideMode = 'sample'; 
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;
net.trainParam.max_fail = 6;
net.trainParam.epochs = 1000;
net.performFcn = 'crossentropy'; 
[net,tr] = train(net,x,t);
y = sim(net,x);
PredProb_NN = transpose(y(2,tr.testInd));
testInd(tr.testInd) = true;
[XNN,YNN,~,AUC_NN] = perfcurve(TrainObsNN(testInd,end),PredProb_NN, posclass);
display(sprintf('Method: patternnet (#1): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr.best_epoch,tr.best_perf,AUC_NN)
%
%
%%Neural Networks feedforwardnet #1
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net2 = feedforwardnet(hiddenLayerSize,trainFcn);
net2.performFcn = 'crossentropy';
net2.outputs{numel(hiddenLayerSize)+1}.processParams{2}.ymin = 0;
net2.layers{net2.numLayers}.transferFcn = 'softmax';
net2.plotFcns = {'plotperform','plottrainstate','ploterrhist','plotconfusion','plotroc'};
net2.initFcn='initlay';
net2.layers{1}.initFcn='initnw';
net2.layers{2}.initFcn='initnw';
net2.layers{1}.transferFcn = 'tansig';
net2 = init(net2);
net2.input.processFcns = {'removeconstantrows','mapminmax'};
net2.output.processFcns = {'removeconstantrows','fixunknowns'};
net2.divideFcn = 'divideblock';  
net2.divideMode = 'sample';  
net2.divideParam.trainRatio = 60/100;
net2.divideParam.valRatio = 20/100;
net2.divideParam.testRatio = 20/100;
net2.trainParam.max_fail = 6;
net2.trainParam.epochs = 1000;
[net2,tr2] = train(net2,x,t);
y2 = sim(net2,x);
PredProb_NN2 = transpose(y2(2,tr2.testInd));
testInd2(tr2.testInd) = true;
[XNN2,YNN2,~,AUC_NN2] = perfcurve(TrainObsNN(testInd2,end), PredProb_NN2, posclass);
display(sprintf('Method: feedforwardnet (#1): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr2.best_epoch,tr2.best_perf,AUC_NN2))
%
%
%%Neural Networks patternnet #2
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net3 = patternnet(hiddenLayerSize, trainFcn);
net3.input.processFcns = {'removeconstantrows','mapminmax'};
net3.output.processFcns = {'removeconstantrows','fixunknowns'};
net3.divideFcn = 'divideblock';  
net3.divideMode = 'sample'; 
net3.divideParam.trainRatio = 60/100;
net3.divideParam.valRatio = 20/100;
net3.divideParam.testRatio = 20/100;
net3.trainParam.max_fail = 6;
net3.trainParam.epochs = 1000;
net3.performFcn = 'crossentropy';
[net3,tr3] = train(net3,x,t);
y3 = sim(net3,x);
PredProb_NN3 = transpose(y3(2,tr3.testInd));
testInd3(tr3.testInd) = true;
[XNN3,YNN3,~,AUC_NN3] = perfcurve(TrainObsNN(testInd3,end), PredProb_NN3, posclass);
display(sprintf('Method: patternnet (#2): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr3.best_epoch,tr3.best_perf,AUC_NN3))
%
%
%%Neural Networks feedforwardnet #2
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net4 = feedforwardnet(hiddenLayerSize,trainFcn);
net4.performFcn = 'crossentropy';
net4.outputs{numel(hiddenLayerSize)+1}.processParams{2}.ymin = 0;
net4.layers{net4.numLayers}.transferFcn = 'softmax';
net4.plotFcns = {'plotperform','plottrainstate','ploterrhist','plotconfusion','plotroc'};
net4.initFcn='initlay';
net4.layers{1}.initFcn='initnw';
net4.layers{2}.initFcn='initnw';
net4.layers{1}.transferFcn = 'tansig';
net4 = init(net4);
net4.input.processFcns = {'removeconstantrows','mapminmax'};
net4.output.processFcns = {'removeconstantrows','fixunknowns'};
net4.divideFcn = 'divideblock';  
net4.divideMode = 'sample';  
net4.divideParam.trainRatio = 60/100;
net4.divideParam.valRatio = 20/100;
net4.divideParam.testRatio = 20/100;
net4.trainParam.max_fail = 6;
net4.trainParam.epochs = 1000;
[net4,tr4] = train(net4,x,t);
y4 = sim(net4,x);
PredProb_NN4 = transpose(y4(2,tr4.testInd));
testInd4(tr4.testInd) = true;
[XNN4,YNN4,~,AUC_NN4] = perfcurve(TrainObsNN(testInd4,end), PredProb_NN4, posclass);
display(sprintf('Method: feedforwardnet (#2): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr4.best_epoch,tr4.best_perf,AUC_NN4))
%
%
if ~isequal(tr.testInd,tr2.testInd) || ~isequal(testInd,testInd2) || ...
      ~isequal(tr.testInd,tr3.testInd) || ~isequal(tr.testInd,tr4.testInd)
  warning('The test sets are not the same!')
end
if isequal(y,y3) && isequal(y2,y4)
 display('Random number generation is *not* an issue.')
else
  warning('There is a random number generator issue')
end
if isequal(y,y2) || isequal(y3,y4)
 display('The two networks are identical')
else
  warning('The two netowrks are *not* identical!')
end

Output:

Method: patternnet (#1): Best Epoch: 122, Best performance: 0.2379, AUC: 0.7348

Method: feedforwardnet (#1): Best Epoch: 136, Best performance: 0.2380, AUC: 0.7331

Method: patternnet (#2): Best Epoch: 122, Best performance: 0.2379, AUC: 0.7348

Method: feedforwardnet (#2): Best Epoch: 136, Best performance: 0.2380, AUC: 0.7331

Random number generation is not an issue. Warning: The two netowrks are not identical!

Answer 1

[Greg Heath]

Why didn't you just make a simple search on

 patternnet feedforwardnet

and find

 http://www.mathworks.com/matlabcentral/answers/196799-how-to-simulate-default-patternnet-with-feedforwardnet-in-matlab

Thank you for formally accepting my answer

Greg