Help using perfcurve function!!

Hey everyone,

I have three groups of people normal ones, one with disease in its early stages and last one people with disease in its severe stages. And in each group I study 16 different features. In order to check the discriminatory performance of several features, I need to use area under the receiver operating characteristic curve. I don't know how can i translate my data to input arguments for the perfcurve function!! [X,Y,T,AUC,OPTROCPT,SUBY,SUBYNAMES] = perfcurve(labels,scores,posclass);

Any help will be appreciated.

[Yasmin Tamimi]

here is my code but even after writing it and getting the results I still don't understand what does the numbers mean or interpret for X, Y, and AUC!! and on what bases did we choose to assign the ones and zeros for the labels and if I flip will I get different results!! plz advice

labels = {'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'can-';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'ecan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+';'dcan+'};

data= [Data_normal; BB; CC ]; % the first group for normal people then people with early disease and finally with severe disease

x1= data((1:132),:); %combinations of can- and ecan+

x2= data([1:70 133:144],:); %combinations of can- and dcan+

x3= data((71:end),:); %combinations of ecan+ and dcan

y1 = (1:132)'>70; %% can- =0, ecan+ =1

y2 = (1:82)'>70; %% can- =0, dcan+ =1

y3 = (1:74)'>62; %% ecan+ =0, dcan+ =1

b1 = glmfit(x1,y1,'binomial');

b2 = glmfit(x2,y2,'binomial');

b3 = glmfit(x3,y3,'binomial');

p1 = glmval(b1,x1,'logit');

p2 = glmval(b2,x2,'logit');

p3 = glmval(b3,x3,'logit');

[X1,Y1,T1,AUC1] = perfcurve(labels((1:132),:),p1,'ecan+');

[X2,Y2,T2,AUC2] = perfcurve(labels([1:70 133:144],:),p2,'dcan+');

[X3,Y3,T3,AUC3] = perfcurve(labels((71:end),:),p3,'dcan+');

figure(1)

plot(X1,Y1)

xlabel('False positive rate'); ylabel('True positive rate')

title('ROC for classification by logistic regression')

figure(2)

plot(X2,Y2)

xlabel('False positive rate'); ylabel('True positive rate')

title('ROC for classification by logistic regression')

figure(3)

plot(X3,Y3)

xlabel('False positive rate'); ylabel('True positive rate')

title('ROC for classification by logistic regression')

Answer 1

[Ilya]

Other than a few typos (for example, y1 = (1:132)'>70; clearly is not what you meant), I don't see anything wrong with your code.

glmfit/glmval is your classifier.

The choice of the positive and negative classes is arbitrary. You can always swap them. By convention, the predicted scores must be large for the positive class and small for the negative class. For example, for the first ROC you choose ecan+ as your positive class. Then, I suppose, you set y1 to 1 for ecan+ and 0 for can-. The response returned by your logistic regression model will then be high for ecan+ and low for can-.

You can find many sources (textbooks, web pages etc) that interpret FPR, TPR and AUC.

[Yasmin Tamimi]

I really appreciate your help and still have some questions. First, why did you think I didn't mean it when I wrote y1 = (1:132)'>70,, I have 70 samples from can-, 62 from ecan+ and 12 from dcan+!! do you think I have to pad them with nans to get them all the same dimensions?! Second, when I say that AUC1 = 0.7191 what does that mean?! and what happens or what physical meaning does it give if I only considered like one feature each time from the 16 - I mean at that time I have to repeat the algorithm 3x16- ?! Finally, what does it mean response returned by your logistic regression model will then be high for ecan+ and low for can- ,, does it mean that the features in ecan+ are more significant than can- ?!!

[Ilya]

Sorry, I didn't look at your code carefully. Your y1 is fine.

The response for the logistic regression at any point is a number between 0 and 1. If it is close to 1, it means ecan+ is more likely; if it is close to 0, it means can- is more likely.

You used all predictors (features) in data in your analysis. Your conclusions, whatever they are, are only valid for all these features included. If you want to drop some of your features, you would have to refit your 3 models and repeat the ROC analysis.

[Yasmin Tamimi]

Ok! now I understood. One final question: Instead of writing y1 in a binary form I tried writing it as a combination of -1 and I ended up with an error bcz in glm I wrote binomial!! so what kind of distribution do you suggest for now!!

[Yasmin Tamimi]

i used normal distribution and it gave me the same results as in binomial!! no difference!!

[Ilya]

Your response values are categorical, which means your problem can be treated as a classification problem. The binomial distribution for glmfit is then the obvious choice.

Statistics Tlbx provides many other tools for classification. You might want to start here:

http://www.mathworks.com/help/toolbox/stats/f15360dfi14.html

http://www.mathworks.com/help/toolbox/stats/bsvjxt5.html

There is no way to predict what model would work best for you. "Best" is usually chosen by balancing the model accuracy and its interpretability.

Answer 2

[Ilya]

This is a rather vague question. You might get more replies if you explain what approaches you are thinking about and what you have tried so far.

The three groups of people define the labels. You must choose one of these groups as your positive class. Scores typically represent model predictions. You could train a classifier, for instance, to separate the 3 classes and use its predictions.

A ROC curve is used for two classes (groups). I don't know what you mean when you say you need AUC for 3 groups.

[Yasmin Tamimi]

i have chosen combinations of two from the three groups,, so i did three different cases!! But what do u mean that I have to train my classifier!! I updated my question and added my code..