Neural Net: Saving Trained Net

I will explain my problem. My code is given below.

% ********************* Plant ************************

 s = 5000;        u    = zeros(1, s); 
 y = zeros(1, s); yhat = zeros(1, s);

for k=1:s 
    u(k)= unifrnd(-1,1); 
    if (k==1), 
        y(k)=0; 
    elseif (k==2), 
        y(k)=0.3*y(k-1)+u(k)+0.3*u(k); 
    else 
        y(k)=0.3*y(k-1)+0.6*y(k-2)+u(k)+0.3*u(k); 
    end 
end

% **************** NN Modelling ********************* % Creating Neural Net

 [yn,ys] = mapminmax(y); 
  net    = newcf( u, yn, [20 10 ], 'tansig', 'tansig',...
           'purelin'},'trainscg');

% Training Neural Net 

 net.trainParam.lr = 0.05; net.trainParam.lr_inc =1.05;       
 net.trainParam.lr_dec = 0.7; net.trainParam.hide = 50; 
 net.trainParam.mc = 0.9; net.trainParam.epochs = s;
 net.trainParam.goal = 1e-5; net.trainParam.max_fail = s;
 net.trainParam.time = 3*3600; trainInd = 1:1:s; 
 valInd = 2:50:s; testInd = 3:50:s; 
 [trainu,valu,testu] = divideind,u,trainInd,valInd,testInd);
 [trainy,valy,testy] = divideind(yn,trainInd,valInd,testInd); 
 net = init(net); net = train(net,u,yn);

The actual problem is different. The above program is written in an m-file and takes about 15 minutes to converge. Can you go through the program and tell me if I am correct in programming? I wish to use this net for another set of inputs. For this, I need to access the trained net. How the trained net is saved external to this m file and how to access it? Please help. Thanks in advance.

[Walter Roberson]

If you want someone to go through your code, I recommend that you space it out to make it readable instead of packing as much as possible on one line.

Answer 1

[Greg Heath]

I am not sure why you used a cascade forward net. That net has no feedack loops.

The generating equations indicate that a narxnet is more appropriate.

Is u(k)+ *0.3*u(k) a misprint? Should the latter be u(k-1)?

When you change to a narxnet, check the significant values (>0.21) of the crosscorrelation function nncorr(U,Y,N-1)and the autocorrelation function nncorr(Y,Y,N-1) to determine good values for the delays. The capitals indicate zscore transformations (i.e., U = zscore(u), etc).

Also change the divide function from dividerand (destroys u-y and y-y correlations) to either 'divideblock' or another choice.

As for your CFnet, you have 2 hidden layers with 30 nodes resulting in Nw = (1+1)*20+(20+1)*10+(10+1)*1 = 261 weights. A narxnet will probably need fewer hidden nodes and weights.

Hope this helps.

Thank you for accepting my answer.

Greg

[Unnikrishnan PC]

Dear Greg,

Thank you for your valuable suggestions. The current value of the output depends on a non linear function of the input and past values of output. later I may add input delays too. Do you suggest NARXNET for these type of problems? How can I check the significant values (>0.21) of the crosscorrelation function nncorr(U,Y,N-1)and the autocorrelation function? Please reply. Thanks for taking out your valuable time. See code below.

for k=1:s

     u(k)= unifrnd(-1,1);
     if (k==1) 
         y(k)=0;
        elseif (k==2) 
            y(k)=0.3*y(k-1)+u(k)^3+0.3*u(k)^2-0.4*u(k);
            else y(k)=0.3*y(k-1)+0.6*y(k-2)+u(k)^3+0.3*u(k)^2-0.4*u(k);
     end
end

[Greg Heath]

Since your input is random, positive input delays should only be considered if those input delays are expicitly in your I/O equation OR you replace the random input with a determininistic input that has significant positive delay correlations.

Greg

Answer 2

[Greg Heath]

If you do not use NARXNET you will have to add additional dimensions to the input matrix to accomodate the target delays. After training, however, you will not have a net that can be readily implemented on unseen data because it will not accomodate feedack delays.

In your case you already know, from the generating equation, that the input delay is 0 and the feedback delays are 1:2. However, you should still

1. Remove the negative delay components autocorrT(1:N-1) = [], etc

2. Plot the resulting nonnegative delay functions autocorrT and crosscorrXT vs nonnegative delays lags = 0:N-1

3. Obtain the indices for significant correlations using the function FIND.

4. Ignore autocorrT(1) and choose a reasonable number of significant positive delays.

5. The smallest practical number of hidden nodes can be obtained by trial and error.

6. Overwrite the correlation-destroying 'dividerand' with an alternative.

7. Assuming T = zscore(y). The post-training values for the nontraining coefficients of determination (see wikipedia) are

R2val = 1 - tr.vperf(end)

R2tst = 1 = tr.tperf(end)

These mean more to me than correlation coefficients that ignore vertical shifts of the linear output/target line fit.

You may find it useful to search the NEWSGROUP and ANSWERS using the searchwords narx and narxnet.

Hope this helps.

Greg

[Unnikrishnan PC]

Thank you Greg, I will implement it and come back soon. Thanks for your valuable comments

Answer 3

[Greg Heath]

%************************************** NN Modelling ********************************

% Creating Neural Net

% [yn,ys] = mapminmax(y);

Better to use "t" for targets and "y" for outputs

Why not transform u? TANSIG works better with bipolar inputs.

% net = newcf(u,yn,[20 10] ,{'tansig','tansig','purelin'},'trainscg');

Probably do not need 2 hidden layers or that many hidden nodes.

% % Training Neural Net

% net.trainParam.lr = 0.05;

% net.trainParam.lr_inc = 1.05;

% net.trainParam.lr_dec = 0.7;

% net.trainParam.hide = 50;

% net.trainParam.mc = 0.9;

The above parameters should not be used with TRAINSCG.

help trainscg

doc trainscg

Why couldn't you just accept the net default parameters??

% net.trainParam.goal = 1e-5;

MSEgoal = 0.01*var(yn,1) = 0.01 is usually sufficient

% net.trainParam.epochs = s;

Why replace the default = 1000 with 5000 ??

% net.trainParam.max_fail = s;

Why replace the default 6 with 5000 ??

% net.trainParam.time = 3*3600;

Why?

% trainInd = 1:1:s;

% valInd = 2:50:s;

% testInd = 3:50:s;

No. They should have no points in common.

% [trainu,valu,testu] = divideind(u,trainInd,valInd,testInd);

% [trainy,valy,testy] = divideind(yn,trainInd,valInd,testInd);

No. Dividing u is sufficient (y is automatically paired with u point-by-point)

% net = init(net);

Unnecessary. NEWCF is self-initializing.

%net = train(net,u,yn);

[ net tr Y E ]= train(net,u,yn);

tr contains all the important info

Hope this helps.

Greg

P.S. You have probably guessed by now that the reason why your program ran about 20 times longer than it should have is because of the defaults you changed.

[Unnikrishnan PC]

Thank you very much for the valuable comments.

[Unnikrishnan PC]

Dear Greg, I have incorporated your suggestions. Accepted default paraeters.Used only one hidden layer.you suggested to transform input too. But it is already in [-1,1] range! Please answer these queries too.

1. How many input patterns should I apply in this case. i.e. what should be the value of s? 2. How many neurons in the hidden layer?

The network does not train properly. The output of the net is not close to the expected output for a test signal. What could be the reason? Program appended here. Can you take out few minutes and run it? Thanks in advance.

clc, clear all; s=1000; x = zeros(1, s); t = zeros(1, s); that = zeros(1, s); %Initializing Arrays

for k=1:s

     x(k)= unifrnd(-1,1);
     if (k==1) 
         t(k)=0;
        elseif (k==2) 
            t(k)=0.3*t(k-1)+x(k)^3+0.3*x(k)^2-0.4*x(k);
            else t(k)=0.3*t(k-1)+0.6*t(k-2)+x(k)^3+0.3*x(k)^2-0.4*x(k);
     end
end
% ***************************************NN Modelling******************** 
% Creating Neural Net  
[tn,ts] = mapminmax(t); %Pre-processing target,t, in the range(-1,1)
[xn,xs] = mapminmax(x); %Pre-processing input,x, in the range(-1,1)
net = newnarx(xn,tn,0,1:2,21,{'tansig','tansig','purelin'},'trainscg');% Create Net

[trainx,valx,testV,trainInd,valInd,testInd] = divideblock(x,0.6,0.2,0.2);

[trainT,valT,testT] = divideind(x,trainInd,valInd,testInd); net.divideFcn='divideblock';

%Training & Saving Net

net = train(net,xn,tn); save net;

%****************Testing net with a sinusoidal input*********************

for k=1:250

     x(k)= sin(2*pi*k/250); %Test Input
     if (k==1)              %Creating actual output for the test input
         t(k)=0;
        elseif (k==2) 
            t(k)=0.3*t(k-1)+x(k)^3+0.3*x(k)^2-0.4*x(k);
            else t(k)=0.3*t(k-1)+0.6*t(k-2)+x(k)^3+0.3*x(k)^2-0.4*x(k);
     end
 end
[tn,ts] = mapminmax(t); [xn,xs] = mapminmax(x);
 for k=1:250
     that(k)=sim(net,xn(k)); %Generating net output for the test input
     plot(k,xn(k),'r.-',k,tn(k),'bo-',k,that(k),'m+-'); hold on; 
 end