About Friedman test implementation

Hi,

I found (maybe I am wrong) that the Matlab implementation of the Friedman test does not give me the same results than a "classical" hand-made resolution for a problem.

I have used the dataset of the Table 6 of the paper "Statistical Comparisons of Classifiers over Multiple Data Sets" (Demsar, Journal of Machine Learning, 2006).

Even the mean ranks of the different classifiers (columns) aren't the same. From that moment, chi-square value does not agree, etc...

Does anybody know why are these differences? Best regards.

[the cyclist]

You might have a better chance of someone exploring this question if you pasted the data from that paper here, in a way that someone could just directly put it into MATLAB.

[José Manuel]

You are right.

data=[0.7630, 0.7680, 0.7710, 0.7980; 0.5990, 0.5910, 0.5900, 0.5690; 0.9540, 0.9710, 0.9680, 0.9670; 0.6280, 0.6610, 0.6540, 0.6570; 0.8820, 0.8880, 0.8860, 0.8980; 0.9360, 0.9310, 0.9160, 0.9310; 0.6610, 0.6680, 0.6090, 0.6850; 0.5830, 0.5830, 0.5630, 0.6250; 0.7750, 0.8380, 0.8660, 0.8750; 1 , 1 , 1 , 1 ; 0.9400, 0.9620, 0.9650, 0.9620; 0.6190, 0.6660, 0.6140, 0.6690; 0.9720, 0.9810, 0.9750, 0.9750; 0.9570, 0.9780, 0.9460, 0.9700];

And I add this question:

Regarding to the "two categorical factors, what is the meaning of this phrase os the help? "Friedman’s test is a nonparametric test for data having a two-way layout (data grouped by two categorical factors)." I noticed that all the examples of the help are with only two factors in the row dimension (with several reps)...

Answer 1

[the cyclist]

I notice a couple things. First, that paper is doing the ranking from largest to smallest effect. MATLAB is doing the opposite. This is just a matter of convention, and should not effect the final results. From MATLAB, I get

meanranks: [1.8571 3 2.0714 3.0714]

The paper gives [3.143 2.000 2.893 1.964].

If we "invert" the MATLAB result, it would be

[3.1429 2.0000 2.9286 1.9286].

These results are (almost) the same as the paper.

I'm not sure, but the remaining differences is likely due to how tied ranks are handled. My advice would be to open up the file ( edit friedman ), place a breakpoint at the beginning, and step through the code line-by-line to see how the exact ranks are calculated.

[José Manuel]

OK, thank you very much.

How do you "invert" the MATLAB result?

[the cyclist]

I did it by doing the Friedman test on the negative of the results in the table. So, I did friedman(-data) rather than friedman(data).

[José Manuel]

Thank you very much again.

I was looking for any parameter to tell the friedman function the criterion to assing the ranks.

I have noticed that the small error between the means of the paper [3.143 2.000 2.893 1.964] and the means of Matlab [3.1429 2.0000 2.9286 1.9286] comes from an error in the paper, rankinkg the dataset on the penultimate row, because there is a draw that the authors have not detected. If one replicate the author's error, the result is the same.

And debugging, I also noticed that friedman function uses anova2 function, where the chi stat is calculated. So, I suppose that dissagreements in this value (9.28 in the paper vs 10.31 by Matlab) maybe come from some assumptions that are done (normality...) when actually Friedman test is non-parametric.

Do you agree?

[the cyclist]

No, I do not agree. I believe that MATLAB is applying a correction term that is not in the paper. In the code for friedman() you'll see these lines:

% Get a matrix of ranks. For the unusual case of replicated

% measurements, rank together all replicates in the same row. This

% is the advice given by Zar (1996), "Biostatistical Analysis."

m = X;

sumta = 0;

for j=1:r

jrows = reps * (j-1) + (1:reps);

v = X(jrows,:);

[a,tieadj] = tiedrank(v(:));

m(jrows,:) = reshape(a, reps, c);;

sumta = sumta + 2*tieadj;

end

I am not familiar the reason this is necessary, but if you look at the calculation of chistat inside friedman(), you will see this correction applied when "sumta" is not equal to zero. That is what makes the MATLAB result different from the paper.

[José Manuel]

You are right. I also do not know why Matlab does that.

Thank you very much.