statistical significance at the 95% confidence level

Hello,

I have two learning algorithm LA1 and LA2. I ran these two learning algorithms on 5 datasets and got 10 numbers (5 for LA1 and 5 for LA2), each shows the average error rate on 10-fold cross-validation.

Someone asked me that I should do the statistical significance for the 95% confidence level. I do not know what it is. Could you please let me know if there is a good text\source to read about it. Also, I would like to know how should I do this test using Matlab.

Can one explain it to me with one example? Let say, for example, the average error of 10fold cross-validation on the 5-datasets using LA1 are: [4.3; 5.6; 30.7; 8.4; 6.4] and for LA2 are: [4.9; 5.8; 30; 8; 7.4]

Thanks

Answer 1

[Ilya]

The reference is

T. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923, 1998.

The implementation can be found in the testckfold function in the Statistics and Machine Learning Toolbox.

[MHN]

Thank you very much for your answer. But I still do not know how to compare them if one of the methods, LA1 or LA2 or both, are not built-in model in Matlab.

[Ilya]

The paper would answer that question for you. The paper is pretty simple. And it takes maybe 10 seconds of your time to google the title and download it.