neural nework validation accuracy calculation

I am currently training a CNN to do modulation classification using NN toolbox. The validation accuracy reported by matlab through the training progress window is approximated 50% at the end of training. But when I tested the accuracy myself using the trained network with the validation data set, it's only about 20% to 25%. This is calculated as (num of correct classification / overall number of tests).

How does matlab calculate the validation accuracy? Where can I find more information? Thanks.

[Greg Heath]

Equations speak louder than words:

   SPEAK!

   o   o
     v
   ~~~~~

Answer 1

[Joy]

sure, here is the code

for snr = snr_str:2:snr_stop

    ct_idx = ct_idx+1;

    %loop through the data to find all signals with snr of current value
    for k = 1:val_len       
        %check if the snr for this entry is correct
        snr_str = string(squeeze(lbl(test_idx(k),2,:)).');
        try
            snr_i = str2num(snr_str); 
        catch 
            snr_i = str2num(char(snr_str));
        end

        if snr == snr_i                 
            [YPre, scores] = classify(TO_net,valImages(:, :, 1, k));

            if isequal(YPre, trainLabels(k))
                %correct prediction
                predictC_cnt = predictC_cnt+1;
            else
                predictW_cnt = predictW_cnt+1;
            end

            ct_idx_c = find(mods_str == string(YPre));
            ct_idx_r = find(mods_str == string(trainLabels(k)));
            confusionTbl(ct_idx, ct_idx_c, ct_idx_r) =  confusionTbl(ct_idx, ct_idx_c, ct_idx_r)+1;
        end 
    end

    predictAcc_i = predictC_cnt/(predictC_cnt+predictW_cnt)*100;
    predictAcc_t(ct_idx) = predictAcc_i;

    %print overall accuracy for current snr
    disp(['overall accuracy for snr = ', num2str(snr), ': ', num2str(predictAcc_i), ...
        '% over ', num2str(predictC_cnt+predictW_cnt), ' tests']);

    total_num_test = total_num_test + predictC_cnt+predictW_cnt;
    predictC_cnt = 0;
    predictW_cnt = 0;

end

[Joy]

Just wondering why the CNN validation accuracy took a dive here on the last iteration? Please see the training progress plot attached.

I intentionally stopped the network from training early by giving a small number of MaxEpochs. This is because I suspected the network was overfitting. But I can't understand why the accuracy dropped so much on the last iteration.

[Don Mathis]

Could you post your network layers and training options?

[Jon Cherrie]

This could be the same issue as this one:

• https://www.mathworks.com/matlabcentral/answers/445800-why-do-i-see-a-drop-or-jump-in-my-final-validation-accuracy-when-training-a-deep-learning-network