How do i train a deep cnn network with multi input with 4 folders of images?

Hi,

I have to train the following network( composed by 4 alexNet pretrained network concatenated into a regression network):

Where every data_1 has an input image [227 227 3].

My idea was this

% IMAGE T/V/T DIVISION
    p='\pato';
    
    trasform=["\cwt" "\hht" "\sswt" "\time"];
    directory = strcat(selpath,'\crossvalidation K=10');
    directory = strcat(directory,'\f');
    directory = strcat(directory,string(k));
    directory_train = strcat(directory,'\train');
    directory_test = strcat(directory,'\test');
    directory_val = strcat(directory,'\val');
    
  
    trainImgsALEX_SQ_cwt = imageDatastore(strcat(directory_train,'\alexsqueeze',p,trasform(1)),'IncludeSubfolders',true,'LabelSource','foldernames');
    trainImgsALEX_SQ_hht = imageDatastore(strcat(directory_train,'\alexsqueeze',p,trasform(2)),'IncludeSubfolders',true,'LabelSource','foldernames');
    trainImgsALEX_SQ_sswt = imageDatastore(strcat(directory_train,'\alexsqueeze',p,trasform(3)),'IncludeSubfolders',true,'LabelSource','foldernames');
    trainImgsALEX_SQ_time = imageDatastore(strcat(directory_train,'\alexsqueeze',p,trasform(4)),'IncludeSubfolders',true,'LabelSource','foldernames');

    
    trainImagesALEX_SQ = combine(trainImgsALEX_SQ_cwt,trainImgsALEX_SQ_hht,trainImgsALEX_SQ_sswt,trainImgsALEX_SQ_time);

    
    testImgsALEX_SQ_cwt = imageDatastore(strcat(directory_test,'\alexsqueeze',p,trasform(1)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');
    testImgsALEX_SQ_hht = imageDatastore(strcat(directory_test,'\alexsqueeze',p,trasform(2)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');
    testImgsALEX_SQ_sswt = imageDatastore(strcat(directory_test,'\alexsqueeze',p,trasform(3)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');
    testImgsALEX_SQ_time = imageDatastore(strcat(directory_test,'\alexsqueeze',p,trasform(4)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');

    testImgsALEX_SQ=combine(testImgsALEX_SQ_cwt,testImgsALEX_SQ_hht,testImgsALEX_SQ_sswt,testImgsALEX_SQ_time);

    valImgsALEX_SQ_cwt = imageDatastore(strcat(directory_val,'\alexsqueeze',p,trasform(1)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');
    valImgsALEX_SQ_hht = imageDatastore(strcat(directory_val,'\alexsqueeze',p,trasform(1)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');
    valImgsALEX_SQ_sswt = imageDatastore(strcat(directory_val,'\alexsqueeze',p,trasform(1)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');
    valImgsALEX_SQ_time = imageDatastore(strcat(directory_val,'\alexsqueeze',p,trasform(1)),'IncludeSubfolders',true,'LabelSource','foldernames','FileExtensions','.jpg');
    
    valImgsALEX_SQ=combine(valImgsALEX_SQ_cwt,valImgsALEX_SQ_hht,valImgsALEX_SQ_sswt,valImgsALEX_SQ_time);
   
   

all of this is inside a for k=1:10 for the k-fold cross validation, and the dataset for training,test, val is divided(for every fold) into 4 folders of images.

Now, if i train the network with the combinedatastore, will this work?If that can be done.

Otherwise, how can i train it in order to have multiple input images from these folders?

Also, I'm having a problem on the concatenation layer

How can i overcome this?

Feel free to propose your way of solving this, doesn't have to be this way only.

Thanks in advance.

Answer 1

[V Sairam Reddy]

Hi Christian,

I understand that you want to train a multi input deep learning network with 4 inputs.

In general, the workflow is to specify a network architecture using a 'layerGraph' object and then train the network using 'trainNetwork' function with datastore input.

With the datastore, you can use the 'combine' and 'transform' functions to create a cell array with 5 columns, where the first 4 columns specify the predictors for each input and the last column specifies the responses. The order of inputs is given by the "InputNames" property of the layer graph layers.

Please find this example to understand the workflow better:

https://www.mathworks.com/help/deeplearning/ug/train-network-on-image-and-feature-data.html

Please find the following resources explaining the training of multi-input network without datastores:

https://www.mathworks.com/matlabcentral/answers/489864-how-to-train-deep-learning-network-with-multi-inputs

https://www.mathworks.com/matlabcentral/answers/522086-how-can-i-train-multi-input-deep-network-without-datastore

https://www.mathworks.com/matlabcentral/fileexchange/74760-image-classification-using-cnn-with-multi-input-cnn?s_tid=srchtitle_multi%2520input%2520cnn_1

I hope this helps.