Create confusion matrix from LDA model

It is easy to train an LDA model and find its accuracy by cross-validation as below:

Mdl = fitcdiscr(data, "Response_var_name", CrossVal="on");

validationAccuracy = 1 - kfoldLoss(Mdl, 'LossFun', 'ClassifError');

However, what is the easiest/best way to get the confusion matrix?

Thanks.

Answer 1

[the cyclist]

The ClassificationDiscrimant class has a predict function. You can input the predicted and actual labels into the confusionchart function.

[Leon]

Thanks. I had considered this but wondered if there was a better way, since accuracy pretty much has its own function (kfoldLoss). I also didn't realise the model stores all of the training data in it; I assumed it was more like what I have found out is called a "compact model" in Matlab. Will cross-validation affect the confusion matrix?

[the cyclist]

Yes, cross-validation can affect how you handle the confusion matrix. And, in some ways you cannot have both the "easiest" and "best" way to do this.

I have typically used ensemble models (with cross-validation), and in the end use the selected "best" model, and its prediction. That is perhaps the easiest approach.

But, each fold actually has its own prediction. So, you could calculate a confusion matrix for each fold. Inspecting multiple confusion matrices can actually be a good way to get a feel for how much uncertainty (due, e.g. to sampling error) you have in that estimate. This is actually true of all the statistics that come out of a machine learning model, such as the accuracy, F1 score, etc.

So, in the end, you are left with how to "average" those statistics over the folds. It's all a bit tricky, and I am not sure if there is a canonical or even a conventional way to do that.

[Leon]

Good to know. Maybe I would be better to use kfoldPredict(), then?

[the cyclist]

Yes, I think that is sensible.

I have to admit, though, that I don't fully comprehend how kfoldPredict goes from this statement (from the documentation)

========================================================================

"For every fold, kfoldPredict predicts class labels for validation-fold observations using a classifier trained on training-fold observations."

========================================================================

-- to a single prediction for the model (as opposed to a prediction per fold, which is how I read that statement). It is presumably possible to use the debugger to step into the function and see exactly what it is doing, but I have not done that.