If ntstool use test set values to predict further values?

Hi !

I use Matlab ntstool toolbox to predict time series forecasting. Ntstool gives me suspiciously good results. For example for load ice_data and train,valid,test ratio 0.55/0.15/0.3 show me MSE equal to (more or less) 0.015 when I try to forecast using Neural Network manually I get MSE qual to (more or less) 2.45. Similarly using ARIMA method I MSE 1.3. Error of ARIMA depends of course of parameters of the model but I would like to present order of magnitude.

On this background results returned by ntstool look very suspicious. This is why I would like to ask the question if ntstool use test set values to predict further values and calculate MSE? I mean.. do ntstool predict values that way that:

1. Use neural network to predict value, then put this value to neural network as an input, then predict another value, put predicted value to neural network as an input and repeat that process till the end. or

2. Use neural network to predict value, then put as an input CORRESPONING value from TEST SET to neural network as an input, then predict another value, put CORRESPONING value from TEST SET to neural network as an input and repeat that process.

In first case error accumulate from first prediction to the last otherwise in second case input is updated to correct value on every step. Which method use by default nstool ?!

Thanks in advance for replying :) ! Janek

Answer 1

[Greg Heath]

Since MSE is scale dependent, it is better to report the normalized value NMSE and or the coefficient of determination (AKA Rsquare). See a statistic reference e.g., Wikipedia for more clarity

Average Target Variance (Minimum MSE when the output is constant)

 MSE00 = mean(var(target',1))

Normalized MSE

 NMSE  = MSE/MSE00

Coefficient of variation (Rsquared), the fraction of MSE00 that is modeled by the net.

 Rsq   = 1 - NMSE

I think you are way off course. Try the command line approach.

See the documentation examples

 help narnet
 doc narnet

Then search ANSWERS and the NEWSGROUP for my tutorial posts. Start with search words

 greg narnet

 greg narnet tutorial

Hope this helps.

Thank you for formally accepting my answer

Greg

[Jan Kostrzewa]

Thanks for you answer but I was asking if ntstool:

1. use predicted value to predict next numbers

or

2. on every step predicted number is update to corresponding value from test set.

Doesn't metter if we use MSE, NMSE or Rsq if we use 1 method we get much higher error then in case of 2 method :)

[Jan Kostrzewa]

I made some research and it appears that, Matlabcode on every step predicted number update prediction to corresponding value from test set.

This is the difference between test set and horizon. In case of horizon we made multistep prediction in case of test set we made one-step prediction many times :)