Spliting ground truth data into 70% training, 20% validation and 10% Testingdata

Hi guys

i am looking for spliting data from ground truth file for following code

70% training, 20% validation and 10% Testingdata

load('gTruth.mat')
Fruits = selectLabels(gTruth,{'orange','Apple'});
if isfolder(fullfile('ExperimentData'))
    cd ExperimentData
else
     mkdir ExperimentData
end 
  
 
addpath('ExperimentData');
splitFolder = true;


if splitFolder
    valFolderName = "ExperimentData";
    
    files = dir(fullfile(valFolderName));
    N = length(files);
    tf=randperm(N)>(0.20*N);
    mkdir TrainingData;
    mkdir ValidationData;
    mkdir TestingData;

    for i=3:length(tf)
        files_re = files(i).name;
        if tf(i) == 3
            copyfile (fullfile(valFolderName,files_re),'TrainingData')
        else
            copyfile (fullfile(valFolderName,files_re),'Validation')
        ifelse
            copyfile (fullfile(valFolderName,files_re),'TestingData')

        end
    end
end

Answer 1

[Shantanu Dixit]

Hi Abdussalam,

To split your ground truth data into training, validation, and testing sets, you can use the 'randperm' function to randomly shuffle the data and 'copyfile' to copy the data from the original data to the corresponding splits.

You can refer to the below code snippet to split data in 70:20:10 ratio

% Generate a random permutation of indices
% N = total data points
indices = randperm(N);

% Determine the number of files for each split
numTrain = round(0.7 * N);
numVal = round(0.2 * N);
numTest = N - numTrain - numVal;


mkdir('TrainingData');
mkdir('ValidationData');
mkdir('TestingData');

% Assuming ExperimentData contains the data files
for i = 1:numTrain
    copyfile(fullfile('ExperimentData', files(indices(i)).name), 'TrainingData');
end

for i = numTrain+1:numTrain+numVal
    copyfile(fullfile('ExperimentData', files(indices(i)).name), 'ValidationData');
end

for i = numTrain+numVal+1:N
    copyfile(fullfile('ExperimentData', files(indices(i)).name), 'TestingData');
end

Additionally, you can refer to the following MathWorks documentation for more information:

'round' : https://www.mathworks.com/help/matlab/ref/double.round.html

'randperm' : https://www.mathworks.com/help/matlab/ref/randperm.html