Error with 1dPooling in dlnetwork

Hi,

I'm having the following error when trying to initialize my dlnetwork :

Layer 'Attention_D': Input data must have two spatial dimensions only, one spatial dimension and one temporal dimension, or two spatial dimensions and one temporal dimension. Instead, it has 0 spatial dimensions and 0 temporal dimensions.

I have seen the work around proposed in https://www.mathworks.com/matlabcentral/answers/1935274-i-m-having-trouble-with-convolution1dlayer, that recommends to use 2d pooling with a poolsize of [ D 1 ] instead of 1d pooling with size D. However, this work-around requires to replace the FeatureInputLayer by an ImageInputLayer.

My problem is : my network aims to classify sequential data, and as a result uses a SequentialInputLayer.

I can't figure how to use the work around on it. Is it possible to use an ImageInputLayer in my case ? My sequences have more than 1 feature/channel.

Or is there another way to deal with this error ?

FYI, my layers are as follow :

layers = [ ...
    sequenceInputLayer(numPredictors, Name="Query")
    lstmLayer(numHiddenUnits,OutputMode="last", Name="Encoder")
    lstmLayer(numHiddenUnits,OutputMode="sequence", Name="Decoder")
    averagePooling2dLayer([poolSizeD,1], Name="Attention_D")
    fullyConnectedLayer(numClasses, Name="Classifier")
    softmaxLayer];

[Matt J]

We need to see the output of,

analyzeNetwork(net,X1,...,Xn)

Answer 1

[Taylor]

Have you tried working with the Deep Network Designer and Network Analyzer? They're quite useful for debugging network architecture.

In the attached images, you can see that the averagePooling1dLayer is what you want to use. The dimensionality in the name of the average pooling layers refers to the nature of the data coming in via the input layer. Image data is generally referred to as 2D (height x width x numChannel) whereas sequence/signal data is generally referred to as 1D (sequence length x numChannels), omitting the "numChannel" dimenion that occurs in both of them. In the attached images, you can see that I'm assuming sequence input with 10 dimensions or 10 sequences input to the network, but using the averagePooling1dLayer does not yield an error as the averagePooling2dLayer does.

[Matt J]

I think you forgot to attach your image. I am getting errors with the following, however, which seems to be more or less what you described.

%(R2024a)

numPredictors=10;
numHiddenUnits=10;
poolSizeD=3;
sequenceLength=15;
numClasses=5;

layers = [ ...
    sequenceInputLayer(numPredictors, Name="Query")
    lstmLayer(numHiddenUnits,OutputMode="last", Name="Encoder")
    lstmLayer(numHiddenUnits,OutputMode="sequence", Name="Decoder")
    averagePooling1dLayer(poolSizeD, Name="Attention_D")
    fullyConnectedLayer(numClasses, Name="Classifier")
    softmaxLayer];

analyzeNetwork(layers,...
    dlarray(rand(sequenceLength,numPredictors),'TC'))

[Clara]

Thank you both for your answers,

My data is 10*100*128 (C*T*B). One sequence is 100 time steps, and has 10 channels. Is it wrong to think that one sequence is a 2d object, just like an image ? If that's right, I don't see how I could reshape it to better fit the 2dpooling, to me it already looks the right shape ?

Also, I'm sorry to go away from the initial topic, but I don't think I understand how RNN works. Should the number of hidden units in an LSTM layer be the number of time step in a sequence, or the number of sequence in a batch ? When I look online at figure of encoder-decoder, it looks to me that each LSTM in a layer receive one time step of the sequence, but then I see tutorials talking about 3d inputs (CBT) for LSTM layers.

Thanks again for all your help.

[Matt J]

Is it wrong to think that one sequence is a 2d object, just like an image ?

It is not enough that the time points of a sequence be 2D arrays for them to be seen as images. The type of input layer you use determines how the dimensions of the network input are labeled (as S,C,T,B, etc...). Each layer then looks at how the dimensions of its input are labeled and produces an output with a new set of dimension labels depending on whatever transformation the layer is doing. The way the dimension labels evolve through the network can be examined with analyzeNetwork().

You will see from analyzeNetwork that the activation map that reaches your averagePooling layer has no dimensions labeled as "S" (spatial), which is what it would need to have in order to be interpreted as an image. In my answer below, I propose that you force the network to re-interpret one or more of the dimensions as spatial (S) using a functionLayer, but I'm not sure if @Taylor would agree that that is necessary.

[Taylor]

Looks like I reached my attachment limit and will have to wait 24 hours to attach the images.

I overlooked this earlier, but the reason you are getting the error is that you are setting the first LSTM layer to the output mode "last". This is related to data format/"dimension label" that @Matt J was mentioning. Here is a page that talks about data formats for deep learning in more detail. Setting the ouput mode to "last" for the LSTM layer outputs the last step of the sequence, eliminating the Time dimension. The Average Pooling 1D layer averages over the Time dimension for sequences. Matt's approach is essentially reformatting the data so the Average Pooling 1D layer recognizes it as 1D image data and averages over the Spatial dimension.

My question is whether you are trying to classify each time point of the sequence or the sequence as a whole. In the former case, you will use the output mode "last" from the LSTM layer, and in the latter, you will use "sequence".

[Clara]

I was trying to classify the sequence as a whole, so the "last" was in fact the issue here. Thanks a lot !

Answer 2

[Matt J]

You can use functionLayers to reformat the activations as the averagePooling layers require, but make sure you set the spatial dimension S to the dimension you intend to pool.

numPredictors=10;
numHiddenUnits=14;
poolSizeD=3;
sequenceLength=20;
numClasses=2;
numBatch=5;

layers = [ ...
    sequenceInputLayer(numPredictors, Name="Query")
    lstmLayer(numHiddenUnits,OutputMode="last", Name="Encoder")
    lstmLayer(numHiddenUnits,OutputMode="sequence", Name="Decoder")

    %reformat activation as spatial-batch-channel (SBC)
    functionLayer( @(x)dlarray(x, "SBC"),  Formattable=true)

    averagePooling1dLayer(poolSizeD, Name="Attention_D")
    fullyConnectedLayer(numClasses, Name="Classifier")
    softmaxLayer];

analyzeNetwork(layers,...
    dlarray(rand(sequenceLength,numPredictors,numBatch),'TCB'))