Neural Networks extrapolation (using closed network - multistep prediction) can not even predict a line ?

Hi; Is it possible to predict EFFECTIVELY into the future using NAR neural network. I used Neural Networks to predict a simple line using multisteps (closed loop ), but it turns out that it is very good but only till the training part whereas it miserably fails in the multistep (predict ahead ) part. here is my code: Please help, where am I wrong ? %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% MULTISTEP PREDICTION USING NAR AND CLOSED LOOP NEURAL NETWORK

clc; clear all; close all;

% A simple line which I want to predict

DATA= (1:1000)';

%% Create a Nonlinear Autoregressive Network

feedbackDelays = 1:5;

hiddenLayerSize = 5;

net = narnet(feedbackDelays,hiddenLayerSize);

net.divideParam.trainRatio = 85/100;

net.divideParam.valRatio = 15/100;

net.divideParam.testRatio = 0/100; % NO TEST DATA as I only want to test the multistep part in future

net.divideFcn = 'dividerand'; % Divide data randomly

% TRAINING

N=500; % Number of bars used for training

TDATA = DATA(1:N); % Training Data

T = tonndata(TDATA,false,false);

[x,xi,ai,t] = preparets(net,{},{},T); % prepare data

[net,tr] = train(net,x,t,xi,ai); % Train the Network

y = net(x,xi,ai);

% MULTISPTEP PREDICTION - Closed Loop Network

MSDATA = DATA(N-feedbackDelays(end):end); % multistep predict ahead data

T = tonndata(MSDATA,false,false);

netc = closeloop(net);

[xc,xic,aic,tc] = preparets(netc,{},{},T);

yc = netc(xc,xic,aic);

%% PLot training and multistep predicted part

hold on;

plot(cell2mat(t)); % targets in training part

plot(cell2mat(y),'r'); % nn results in training part

plot([nan(1,length(t)'),cell2mat(tc)],'g'); % targets in multistep predicted part

plot([nan(1,length(y)'),cell2mat(yc)],'r'); % nn in multistep predicted part

Answer 1

[Greg Heath]

Since the target is a straight line, no hidden layer is used. (In general, I would find the minimum number of hidden nodes via a double-for-loop multiple design search).

Testing the CL net on the OL design data for 500 timesteps results in an increase of normalized- mean-square-error from 8e-32 to 5e24 !!!. However, the monotonically increasing CL NMSE only exceeds 0.01 after time = 275.

Similarly, testing the CL net on the nondesign prediction data (501 to 1000 ) yields nmse > 0.01 only after time = 757.

 clc; clear all; close all; plt = 0;

 T  = con2seq(1:1000); d=5
 FD = 1:d; H = []; % H changed to prevent overfitting
 neto = narnet(FD,H);
 neto.divideParam.trainRatio = 85/100;
 neto.divideParam.valRatio   = 15/100;
 neto.divideParam.testRatio  = 0/100; % NOTE 
 % neto.divideFcn = 'dividerand'; Default

 Nd = 500          % Design (Train + Val)
 Td = T(1:Nd);     % Design data
 Tp = T(Nd+1:end); % Prediction data
 Np = length(Tp)   % 500

% OPEN LOOP PERFORMANCE

 [Xo,Xoi,Aoi,To] = preparets(neto,{},{},Td); 
 to= cell2mat(To); varto = var(to,1) % 20419
 rng('default')
 [neto,tro Yo Eo Xof Aof] = train(neto,Xo,To,Xoi,Aoi); 
 yo = cell2mat(Yo);
 NMSEo = mse(Eo)/varto % 8.3362e-32

 plt=plt+1, figure(plt)
 subplot(311), hold on
 plot(to,'LineWidth',2)
 plot(yo,'r--','LineWidth',1)
 title('OPEN LOOP PERFORMANCE')

% CLOSED LOOP PERFORMANCE

 [ netc Xci Aci ]  = closeloop(neto,Xoi,Aoi);
 [ Xc Xci Aci Tc ] = preparets(netc,{},{},Td);
 tc = to;          % isequal(Tc,To) = 1
 [Yc Xcf Acf ]     = netc(Xc,Xci,Aci);
 ec    = cell2mat(gsubtract(Tc,Yc));
 NMSEc = mse(ec)/varto    % 5.3035e+24  WOW!
% Is abs(ec) monotonic?
 u = find(diff(abs(ec) <= 0)) % Empty matrix: 1-by-0
 necsq    = ec.^2/varto;
 cumNMSEc = cumsum(necsq)./(1:Nd-d);
 v        = find(cumNMSEc <= 0.01);
 Nv       = length(v)      % 275 
 NMSEcv   = cumNMSEc(Nv)   % 0.0094645
 yc       = cell2mat(Yc);

 subplot(312), hold on
 plot(tc(v),'LineWidth',2)
 plot(yc(v),'r--','LineWidth',1)
 title('CLOSED LOOP PERFORMANCE')
 subplot(313), 
 plot(ec(v),'LineWidth',2)
 title('CLOSED LOOP ERROR')

% MULTISTEP PREDICTION

 Tm = T(Nd-5:Nd+Nv-1);
 [ Xcm Xcmi Acmi Tcm ] = preparets(netc,{},{},Tm);
 tcm = cell2mat(Tcm); vartcm = var(tcm,1) %  6302
 [Ycm Xcmf Acmf ] = netc(Xcm,Xcmi,Acmi);
 ecm              = cell2mat(gsubtract(Tcm,Ycm));
 NMSEcm           = mse(ecm)/vartcm % 1.3771

% The error amplitude is not monotonic

 u         = find(diff(abs(ecm) <= 0)) %[1,12:19]
 necmsq    = ecm.^2/vartcm;
 cumNMSEcm = cumsum(necmsq)./(1:Nv);
 vm        = find(cumNMSEcm <= 0.01);
 Nvm       = length(vm)       % 257 
 NMSEcmv   = cumNMSEcm(Nvm)   % 0.0090988

 plt = plt+1, figure(plt)
 subplot(211), hold on
 plot(tmc(z),'LineWidth',2)
 plot(ymc(z),'r--','LineWidth',1)
 title('TARGET(Blue) & OUTPUT(Red)')
 subplot(212), 
 plot(emc(z),'LineWidth',2)

Hope this helps.

Thank you for formally accepting my answer

Greg

Answer 2

[Greg Heath]

1. Do you really expect to confidently predict ahead when you use 'dividerand' ???

2. Both training and validation data are used for design. Therefore if you do not have test data you do not have an UNBIASED estimate of future performance.

3. Conclusion

   a. Use a divideFcn which preserves order and constant spacing.
   b. Obtain an unbiased estimate of performance on new data using a test set.

Hope this helps.

Thank you for formally accepting my answer

Greg

P.S. IT IS VERY UNFORTUNATE THAT MATLAB USES DIVIDERAND TO BE THE DEFAULT DIVISION FUNCTION IN TIMESERIES FUNCTIONS

[Muhammad Hasnat]

Thankyou for the response.

I should have mentioned earlier that I have already tried different basic variants, e.g divideblock, different number of layers as well as different number of delays. but the line is never matched effectively in multistep prediction ahead. After repeated training one may get a curve at best. Also, shouldn't 70% be enough to find constant spacing of a line, even if 30% is taken out randomly. I say that also because, results were better with random division of data when predicting a line.

Regards Hasnat

Answer 3

[Greg Heath]

 1. You do not need a hidden layer for a straight line. Therefore, you are overfitting. It is good practice to try to minimize the number of hidden layers and nodes. Otherwise use validation stopping and/or regularization.

 2. Test the CL configuration on the OL design data. If you compute the cumulative MSE, you can get an estimate of an upper bound for how many timesteps you can expect to accurately predict.

Hope this helps.

Greg