Problem with exact replication of maglev narx closed-loop output

I wrote a simple Matlab code to calculate a narx within the maglev example. The goal was to replicate the closed-loop output from the usual Matlab expression yc = netc(xc,xic,aic). My code is within the maglev example and uses the weights and bias' from the trained narx, so there shouldnt be any precision issues with reading weights from another file. The problem is my code does not perfectly replicate the closed-loop output. It is very close, and sometimes spot-on, but doesnt replicate perfectly every time.

Has anyone had this problem? I am not arrogant enough to seriously suspect a problem with Matlab, but I have to ask the question to see if anyone has had this issue. It is likely something silly with my code. I can trim it down to a reviewable size if anyone would like to help me out.

Thanks, Cal

Answer 1

[Calvin]

Problem solved. So I'm writing an answer to my own question for the benefit of readers who may have a similar problem. The key is with how matlab interprets the time order of the input vector.

Sidebar: The deploy solution option in ntstool executes genFunction. The resulting m-file hard-codes the architecture and delay info, and the indexing appears unnecessarily complicated. But it was useful for troubleshooting my own code.

The problem with my stand-alone NARX code was with my incorrect assumption of the time order of the input vector. Here's an example of part of my simple code that calls my function narxann4:

for time=1+D:Ntimesteps % do for all training set timesteps after lag D
  InputVector = Xin(time-inputDelays);  %%%NOTE THE ORDER!!!!
  Ycalc(time) = narxann4(InputVector',Ycalc(time-feedbackDelays),...
                         Bias1,WtsInput,WtsContext,Bias2,WtsL2); 
end

Notice that the InputVector is in reverse chronological order. For example, if feedbackDelays=1:2 and time=3, then InputVector=[Xin(2) Xin(1)]

Once I figured this out, my simple stand-alone code replicated the maglev closed-loop simulation exactly. Hope this helps somebody.

Cal

Answer 2

[Greg Heath]

Closed loop designs have the irritating property of propagating errors.

Typically, just closing an openloop design is not sufficient.

After using the closeloop fuction to convert from openloop, you should train the closeloop net starting with the weights obtained from the openloop design.

Search

 greg closeloop

I probably have some examples in the NEWSGROUP and/or ANSWERS.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Calvin]

Thanks Greg. I have seen your posts regarding retraining after closing the loop. Those have been helpful, as well as many of your other posts. My experience with retraining after closing the loop is that the performance for the training period improves, but the test period gets worse. This is probably problem-specific. However the issue I posted can be restated as:

1. train maglev problem in open-loop mode

2. close the loop and simulate the entire dataset

3. within same m-file recalculate the closed-loop time-series using simple code that uses the weights from step 1 and the basic recursion logic for a narx.

4. compare results from step 3 with step 2. Usually results are very close, but not exact for parts of the time-series.

The above steps are part of the troubleshooting I'm doing. The broader application is more involved with K-fold cross-validation and threshold ANNs. It also has an option to do closed-loop retraining. But my posted problem is narrowly focused on the above steps.

I noticed the maglev problem does not have explicit statements to rescale (-1 to 1) the input or targets prior to training. My simple narx code (step 3) requires this scaling (-1 to 1). I suspect somewhere inside one of the matlab functions the inputs and targets are being rescaled. Is that true?

I can trim my simple m-file down to a reviewable size if you have time to check it out. Alternatively the challenge is simple: Use the maglev example and perform the 4 steps above to see if you get agreement in step 4.

Thanks Greg.

[Greg Heath]

http://www.mathworks.com/matlabcentral/newsreader/view_thread/332147#912806

Hope this helps.

Thank you for formally accepting my answer

Greg

[Calvin]

I appreciate the link Greg, altho I had read it earlier this year. It was previously helpful for my broader design issue. My posted question, however, was a narrowly-focused question regarding the inability to replicate the closed-loop maglev simulation with my own code.

I was planning to send you my simple m-file, but I found a very helpful feature with R2014a ntstool (deploy solution) which generates a matlab function to simulate the output from the closed-loop narx. This code should help me move forward with my trouble-shooting.

I'll be happy to formally accept your answer to this post if you can kindly let me know the answer to my above follow-up question which I have restated here:

I noticed the maglev problem simple script does not have explicit statements to rescale (-1 to 1) the input or targets prior to training. My simple narx code (step 3 above) requires this scaling, and so does the matlab-generated function for deployment. I suspect somewhere inside one of the matlab functions the inputs and targets are being rescaled. If true, do you know which function(s) do this implicit scaling? preparets, train, or perhaps narxnet?

Thanks Greg

[Greg Heath]

Impossible for me to respond when you refer to code which i have not seen. Since there are many ways to approach the problem, I would be wasting my time if i tried to guess exactly what you are trying to do and how you are trying to do it.

However, it shouuld be relatively easy for me to comment on posted code.

[Greg Heath]

>>

 net              = narxnet(10);
 inputprocessfcn  = net.inputs{2}.processFcns
 outputprocessfcn = net.outputs{2}.processFcns

inputprocessfcn =

    'removeconstantrows'    'mapminmax'

outputprocessfcn =

    'removeconstantrows'    'mapminmax'

Hope this helps.

Greg

[Calvin]

Yes. That helps Greg. I forgot I can disable the default minmax scaling preprocessing function by using:

net.inputs{1}.processFcns = {'removeconstantrows'};  % mapminmax removed
net.inputs{2}.processFcns = {'removeconstantrows'};  % mapminmax removed
net.outputs{2}.processFcns = {'removeconstantrows'}; % mapminmax removed

Can you remind me what the first line of the above 3 does? I suspect it may have something to do with open vs closed loop mode.

Thanks! Cal

[Greg Heath]

It is a little confusing. Openloop narx has 2 inputs and 1 output

>> net.outputs

   ans =     []    [1x1 nnetOutput]

Therefore, the only output is net.outputs{2}

>> net.inputs

   ans =  [1x1 nnetInput]
          [1x1 nnetInput]

Therefore, there are 2 inputs ( {1} and {2} ). However

    >> net.inputs{1}

    ans = ...    name: 'x'
                 feedbackOutput: []   %input from exogeneous input, x
    ...

    and

    >> net.inputs{2}

    ans = ...   name: 'y'
                feedbackOutput: 2    %input from 2 output feedback delays

Hope this helps.

Greg