Define a custom input layer in the deep learning toolbox

In the Deep Learning Toolbox, it is possible to define one's own custom output layers and hidden layers. Is there no way to define a custom input layer?

Answer 1

[James Gross]

Hi Matt,

Unfortunately, there is no way to create a custom input layer. However, since 2023b, you can use inputLayer to define a custom format input layer.

Hopefully, this addresses your needs, but if it doesn't, please let us know. We'd love to hear about your use case and see what we can do to help!

Cheers,

James

[Matt J]

Thanks, James. Basically, I was hoping to create a custom input layer that automatically resizes input images to a particular set of spatial dimensions, so that it could accomodate inputs of any size and I would never have to think about it.

I wonder if you would consider reclassifying a resize2dLayer and the like to be in the family of input layers, and make it so that the OutputSize property could be understood to serve as the InputSize property of a normal imageInputLayer.

[James Gross]

Thanks for the extra bit of info Matt!

However, I'm not sure I am 100% following. The InputSize property of imageInputLayer does not restrict the size of image inputs. That is, even if you specify an InputSize of [256, 256, 3], there is nothing stopping you from providing an image of size [128, 128, 3].

Therefore, I would think having an imageInputLayer immediately followed by a resize2dLayer would satisfy your use case here. If this is not the case, could you provide some additional information as to why not?

[Matt J]

That is, even if you specify an InputSize of [256, 256, 3], there is nothing stopping you from providing an image of size [128, 128, 3].

@James Gross Can you elaborate on that? If the inputSize doesn't matter, why is it required and why doesn't the following work?

 dln=dlnetwork([imageInputLayer([256,256,3]), ...
                fullyConnectedLayer(1)]  , ...
                Initialize=false );
 
trainnet(rand(128,128,3),1,dln,'mse',trainingOptions('adam'))

Training stopped: Error occurred

Error using trainnet (line 54)
Execution failed during layer(s) "fc".

Caused by:
Error using dlarray/fullyconnect>iValidateWeights (line 207)
The number of weights (196608) for each output feature must match the number of elements (49152) in each observation of the input data.

[James Gross]

Hi Matt,

Sure thing, happy to elaborate! What I meant was that the imageInputLayer and dlnetwork do not prevent an input of different size than the one specified in the imageInputLayer.

However, subsequent layers may still error if the sizes specified in the imageInputLayer are used to initialize the layer such that the input sizes must be of a certain size. For example, in the particular case provided above, it's erroring because initializing the network with those sizes sets the weights of the fully connected layer to have 196608 (i.e. 256 x 256 x 3 x 1) elements, whereas you have provided an input with 49152 (i.e. 128 x 128 x 3) elements, so it cannot perform the matrix multiplication.

If instead you used a series of convolution layers rather than the fully connected layer, it would not necessarily error because the convolution operation is less sensitive to the input size. For example, modifying the code above to use convolution and global max pooling, leads to it no longer erroring.

 dln=dlnetwork([imageInputLayer([256,256,3]), ...
                convolution2dLayer(3,3)), ...
                globalMaxPooling2dLayer, ...
                fullyConnectedLayer(1)], ...
                Initialize=false );
 
trainnet(rand(128,128,3),1,dln,'mse',trainingOptions('adam'))

Hope this answers your question!

Cheers,

James

[Matt J]

Yes, thanks a lot!

Answer 2

[Aastha]

Hi @Matt J,

As I understand you want to define a custom input layer using the Deep Learning Toolbox that performs the desired operation. You can accomplish this using the following steps:

1) Create a custom input layer class called “newInputLayer” that inherits from the “nnet.layer.Layer” class.

For more information on the “nnet.layer.Layer” class kindly refer to the MATLAB documentation link below:

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layers.html

2) In this class, you need to define the constructor method to set the “inputSize” and the name for the input layer. Implement the “predict” method, which takes an input “X” and applies a transformation function, “yourInputTransformation”, to perform the desired operation in the input layer.

If the input layer includes learnable parameters that need to be updated during training, you can define the “backward” function. The MATLAB code below illustrates this:

classdef newInputLayer < nnet.layer.Layer
    properties
        % Define any properties your layer needs
        InputSize
    end
    
    methods
        function layer = newInputLayer(inputSize, name)
            % Set layer name
            layer.Name = name;
            
            % Set input size
            layer.InputSize = inputSize;
        end
        
        function Z = predict(layer, X)
            % Define the forward operation
            Z = yourInputTransformation(X);
        end
        
        function dLdX = backward(layer, X, Z, dLdZ, memory)
            % Define the backward operation 
        end
    end
end

3) You can then incorporate the “newInputLayer” class into your network architecture.

As an example, create a simple network using the custom input layer, which includes two fully connected hidden layers and an output layer with ReLU activation:

inputLayer = newInputLayer([28, 28, 1], 'custom_input');
 
% Define the network architecture
layers = [
    inputLayer
    fullyConnectedLayer(128, 'Name', 'fc1')
    reluLayer('Name', 'relu1')
    fullyConnectedLayer(64, 'Name', 'fc2')
    reluLayer('Name', 'relu2')
    reluLayer('Name', 'output_relu')
];

Hope this is helpful !

[Matt J]

Thanks @Aastha, but I'm afraid that doesn't work, as demonstrated below. The toolbox distinguishes between input layers, hidden layers, and output layers. So, we would need to inherit from something other than nnet.layer.Layer.

inputLayer = newInputLayer([28, 28, 1], 'custom_input');
 
% Define the network architecture
layers = [
    inputLayer
    reluLayer('Name', 'relu1')
];

dln = dlnetwork(layers),

Error using dlnetwork/initialize (line 600)
Invalid network.

Error in dlnetwork (line 182)
net = initialize(net, dlX{:});

Caused by:
Example inputs: Incorrect number of example network inputs. 0 example network inputs provided but network has 1 inputs including 1 unconnected layer inputs.
Layer 'custom_input': Unconnected input. Each input must be connected to input data or to the output of another layer.