LSTM Python hyperparameters v MATLAB

I am reading a LSTM research paper and it states:

The following experiments investigate deep RNN models parameterized by the following hyperparameters: 1. num_layers – the number of memory cell layers 2. rnn_size – the number of hidden units per memory cell (i.e. hidden state dimension) 3. wordvec – dimension of vector embeddings 4. seq_length – number of frames before truncating BPTT gradient

I can see 2 and 3 as the number of hidden units and the input size but I cannot find where one would set 1 and 4.

Answer 1

[David Willingham]

Hi Philip,

For 1, by default layers are not a "settable" parameter. You need to setup an experiment that tests networks of different sizes and see which one might give the best results. This example Try Multiple Pretrained Networks for Transfer Learning shows how you can use the Experiment Manager App in MATLAB to do this.

For 4, while I don't have an example to share on this. You could use Experiment Manager to setup an experiment that changes the sequence length of the input data used to feed the LSTM training.

[Philip Hua]

hi David,

Thank you for your help and suggestion. The author already tried different settings (in Python RNN) and came up with the the "optimal" set of hyper-parameters. Is there any plan for Mathworks to include these rather basic options?

[David Willingham]

For 1, in MATLAB this isn't a settable parameter, however you can set them manually:

[lstmLayer(64); lstmLayer(64)]

For 4, there is an option to set sequencelength for the mini batch in the trainingoptions:

https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html

Answer 2

[Philip Hua]

Thank you David. Could you however, clarify the suggested network configuration above? The number of memory cells i think is not the same as the number of lstm layers right? Perhaps you could kindly send an unrolled network diagram and label the suggested configuration ?