How to do cross validation for a minibatch queue?

In https://in.mathworks.com/help/signal/ug/signal-source-separation-using-w-net-architecture.html

9 subjects are used for training and 1 subject is used for testing. In this example both training and testing data are saved in different folders. Here input is given to the network as minibatches. I have a similar situation where I need to use cross validation. How to use cross validation when input to the network is as minibatch queue? I need to do the cross-validation in such a way that any one subject can be used for testing and all others are used for training.

Answer 1

[Ganesh]

Hi @BIPIN SAMUEL,

I understand that you would like to set up your system for cross validation when input is given as a minibatch queue.

A simple way you could achieve this is by manipulating the data you feed to the network within a loop. At every iteration you can create a folder structure which contains the "train folder" of the 9 chosen data points and the "test folder" containing the last data point. Here is conceptual working of the same:

mainFolder = 'master_folder_path'; % Replace with the path to your main folder
subfolders = dir(mainFolder);
subfolders = subfolders([subfolders.isdir]);
subfolders = subfolders(~ismember({subfolders.name}, {'.', '..'}));

numSubfolders = numel(subfolders);
combinations = nchoosek(1:numSubfolders, numSubfolders-1); % As we need 1 data for test set, can be modified
tDir = tempdir;

for i = 1:size(combinations, 1)
    combinationFolder = fullfile(tDir, sprintf('Batch_%d', i));
    if ~exist(combinationFolder, 'dir')
        mkdir(combinationFolder);
    end
    
    trainFolders = combinations(i, :); 
    testFolder = setdiff(1:numSubfolders, trainFolders);
    
    for j = 1:numSubfolders
        if ismember(j, trainFolders)
            folderName = 'train';
        else
            folderName = 'test';
        end
        
        subfolderPath = fullfile(combinationFolder, folderName, subfolders(j).name);
        if ~exist(subfolderPath, 'dir')
            mkdir(subfolderPath);
        end
    end

    trainFol = combinationFolder+"\train";
    testFol = combinationFolder+"\test";

    % -----------------------------------
    % Train using the newly created folder 
    % -----------------------------------
    
    rmdir(combinationFolder,'s') % Delete the folder once used
end

Further, you may modify the function "nchoosek()" to use training data of other sizes.

Hope this helps!