ROC curve for multiclass classifier

Hi,

I want to plot RoC curve for multiclass (6 class in total) classifiers that includes SVM, KNN, Naive Bayes, Random Forest and Ensemble. I did calculated the confusion matrix along with Precision Recall but I'm not able to generate the graph that includes ROC and AUC curve. I did tried perfcurve but its for binary class. Can anyone please help me figuring out how to resolve this? Thank you.

Answer 1

[reza]

I am also having the same problem.

Answer 2

[Antonio Mendes]

Same problem...

Answer 3

[Mobayode Akinsolu]

See the following the example for three classes available here (note that you will need to view the section on "Plot ROC Curve for Classification Tree" on the page): https://www.mathworks.com/help/stats/perfcurve.html

For mutiple classes, I will suggest the following according to the example above:

[validationPredictions, validationScores] = validationPredictFcn(validationPredictors)
% validationPredictFcn is any user-defined prediction function which returns the scores - see help info. on "predict" and others.
% validationPredictors is the test data (e.g., table (holding the predictors) of size n-by-k
% n is the number of samples or observations
% k is the number of classes
% validationPredictions is the cell containing all predictions (size is n-by-1)
% validationScores (n-by-k) are the posterior probabilities that an observation (a row vector of predictors) belongs to a class

Assuming that I have four classes ('A', 'B', 'C', and 'D') and I want to plot the ROC curve for the second class which is labelled 'B' (making 'B' the positive class and the other classes, 'A', 'C', and 'D' the negative classes), I will try the following to obtain a score for the positive class 'B' which factors in the negative classes 'A', 'C', and 'D':

diffscore=zeros;
for i=1:size(scores,1)
    temp=scores(i,:); % a row vector holding the scores for the classes [A, B, C, D] for the ith observation out of the total.
    % score of +ve class minus the maximum of the scores of all the negative classes (similar to the example available via the webpage link) 
    diffscore(i,:)=temp(2)-max([temp(1),temp(3),temp(4)]); 
end

Once the new scores have been obtained, you can proceed to plot the ROC curve for the positive B class ('A', 'C' and 'D' are the negative classes) as follows (similar to the example available via the link):

[X,Y,T,~,OPTROCPT,suby,subnames] = perfcurve(validationResponse,diffscore,'B'); % validationResponse: True class labels

"validationResponse" in this case is specified as the cell array of character vectors, or categorical array holding the true class lables for "validationPredictors", the test data.

% Plot ROC Curve
plot(X,Y)
hold on
plot(OPTROCPT(1),OPTROCPT(2),'ro')
xlabel('False positive rate') 
ylabel('True positive rate')
title('ROC Curve')
hold off

You can try the above for as many classes as possible by simply revising this line in the code according to the current positive class and the negative classes. For example:

diffscore(i,:)=temp(2)-max([temp(1),temp(3),temp(4),temp(5),temp(6)]); % for six classes 

Please let me know if this helps.

[Ismat Mohd Sulaiman]

So that means an ROC curve and its corresponding AUC is only applied for one class. If there are 4 classes, there would be 4 ROC curve and corresponding AUC, correct?

[Mobayode Akinsolu]

Hi @Ismat Mohd Sulaiman,

From an anlaytical viewpoint, the number of curves will depend on the sort of comparisons you want to make or report on. E.g., one class v other classes (as I have carried out) or one class v another class.

If you have 4 classes and you are opting for a "one class v other classes" approach as I have described above, you will definitely have 4 ROC curves and associated AUC for the following comparisons:

+Class (One) v -Classes (Two, Three and Four)

+Class (Two) v -Classes (One, Three and Four)

+Class (Three) v -Classes (One, Two and Four)

+Class (Four) v -Classes (One, Two, and Three)

For a "one class v another class" comparison, you will have 12 ROC curves and associtated AUC for the following comparisons:

+Class (One) v -Class (Two)

+Class (One) v -Class (Three)

+Class (One) v -Class (Four)

+Class (Two) v -Class (One)

+Class (Two) v -Class (Three)

+Class (Two) v -Class (Four)

+Class (Three) v -Class (One)

+Class (Three) v -Class (Two)

+Class (Three) v -Class (Four)

+Class (Four) v -Class (One)

+Class (Four) v -Class (Two)

+Class (Four) v -Class (Three)

I hope that helps.

Cheers!

[Peter]

Hi,

Good explanation.

Does anyone know a good reference which validates the diffscore method used in the perfcurve examples?

Peter

[Mobayode Akinsolu]

Hi @Peter,

I reckon that any good mathematics/statistics text focused on probability ought to provide you with a very detailed rationale for using these scores.

For "quick" references:

Please see Section 5.6.5 Area under the ROC Curve (AUC) in: A. Burkov, The Hundred-Page Machine Learning Book, vol. 1. Québec, QC, Canada: Andriy Burkov, 2019.

We have also explained this briefly in the 2nd paragraph of Section VI A, in this fairly recent published paper.

I hope that helps.

[Peter]

Thank you