Closed Loop performance of trained NAR not good enough. Why?

Hello, I'm using NAR to predict future values of my timeseries, but when I plot response of closed loop network I saw output shifts in time, I could not figure out why. Can anybody help me? Here are my codes, basically most of it generated from ntstool, and some are my modifications.

T = normalizedsolarpower;
N = length(T)
trainFcn = 'trainlm';  % Levenberg-Marquardt backpropagation.

% Create a Nonlinear Autoregressive Network
feedbackDelays = 1:2;
hiddenLayerSize = 10;
net = narnet(feedbackDelays,hiddenLayerSize,'open',trainFcn);

% Prepare the Data for Training and Simulation

[x,xi,ai,t] = preparets(net,{},{},T);

% Setup Division of Data for Training, Validation, Testing
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;

% Train the Network
[net,tr] = train(net,x,t,xi,ai);
rng(0)
% Test the Network
y = net(x,xi,ai);
e = gsubtract(t,y);
performance = perform(net,t,y)

% View the Network
view(net)

% Closed Loop Network
netc = closeloop(net);
netc.name = [net.name ' - Closed Loop'];
view(netc)
[xc,xic,aic,tc] = preparets(netc,{},{},T);
yc = netc(xc,xic,aic);
closedLoopPerformance = perform(net,tc,yc)
plotresponse(tc,yc)
ypredic = netc(cell(0,96),xic,aic); % 96 step-ahead predictionwithout any input
figure
plot(cell2mat(ypredic))
legend('predicted values')

As you can see from plot the error is very high and it seems shift in period, and the closedloopperformance equals to 0.0615, which is high I guess. Any help will be appreciated. Thanks,

[dpb]

Nobody will be able to do anything specifically without the data to run the code and investigate.

BUT while the plot is so small it's hard for my old eyes to read, it appears that the majority of the "target" is a rectified sine with at least semi-random excursions on top of it--iow, looks like two separate processes in play.

The trained model appears to have fit the one quite nicely for the most part but doesn't have sufficient terms in it to be able to handle the other.

[Fred]

I posted my data and attached the plot and future 96 step prediction, I hope this time is easy to see. In future prediction plot I have two peaks which does not make sense it is predicting day ahead solar power for clear sky. I do not know what am I doing wrong.

[dpb]

Well, no, not really... :)

I can't tell where the tiny little "target" dots are when there are the other variables at the same point(s). I'd guess just plotting the target alone would help; I presume that would be what you would be trying to model?

[Fred]

Could you please plot tc, and yc as follows, and tell me what do you think?

plot(cell2mat(yc)); hold on
plot(cell2mat(tc)); hold off

I see that magnitude of yc is almost the same for all the time and does not exceeds 0.3 while target (tc) hits 0.9 sometimes.

[dpb]

I don't have NN TB so can't run your code; all I can do is look at the plots you've shown.

What's the real abscissa units for the normalized power? Looks like must be two days from general shape, length()/24/60/60 -> 0.84 so something around 1 sec sampling time?

Just out of curiosity, what's the reason for storing it as a cell array? It is just a double array but

>> whos normalizedsolarpower sp
Name                      Size                 Bytes  Class     Attributes

normalizedsolarpower      1x72867            8744040  cell                
sp                        1x72867             582936  double

>>

where

sp=cell2mat(normalizedsolarpower);

show it takes almost 9 MB to store the same data as is contained in just 600K as a double plus there's the added overhead of dereferencing a cell array to get the content plus the nuisance of having to use the "curlies" instead of just plan parentheses.

While it's not your problem, looks like could improve efficiency quite a bit by just converting to the double array.

[Fred]

It is 15 minute interval time-series, and I stored it in cell because the example datasets are all stored in cell in MATLAB.

Answer 1

[Shashank Bhatia]

For standarized data (mean = 0 & Var = 1), Steps are as follows:

1. Calculate number of hidden layers and number of feedback delays. --> Via autocorrelation method

2. Selection of training function --> depending upon data complexity, length, and time available

3. Divide data between training, validation and Testing --> use divideblock

4. Training the open loop

5. Defining the closed loop network --> choosing appropriate function for closed loop

6. training the closed loop

7. after confirming accuracy (check mse and perform) of the models, use closed loop trained network for prediction.

[Lucas Ferreira-Correia]

Hi when you said

"5. Defining the closed loop network --> choosing appropriate function for closed loop"

do you mean specify a different training solver or what do you mean?

In addition to that does

"6. training the closed loop"

mean that the closed loop should be trained again the exact same way as the open loop, so with the same data?

I realise this is an old thread but your help would be greatly appreaciated!