BERT:how to change the output feature?

https://github.com/matlab-deep-learning/transformer-models

When I use the bert to extract features，the output features are always 768 dimension.How can I change them into 300 dimension?

[Rik]

That is a form of down-sampling. How do you propose to perform this downsampling?

Also, this question doesn't have anything to do with Matlab yet. You could indeed perform this downsampling in Matlab, but the concept itself is a deep learning question, not a Matlab question.

[David Willingham]

Can you explain why you would like to change the output to 300?

[XT]

I want to use the bert to extract features of protein sequence.But when I put the features into the model, I found the model performance is not well. I know that the model performance is affected by many factors, but I want to know whether the features extracted by bert are too long. (Every input sequence is not more than 50 words) So I want to change the output to fewer dimension.

[XT]

I wonder whether can I put a fullyconnected layer to change the output of the bert. If the model is a CNN , I think the fullyconnected layer can work. The bert cant' be seen in the Deep Network Designer, so I don't konw how to change it.

[Rik]

A fully connected layer is equivalent to doing a matrix multiplication. In your case you want a matrix of 768x300.

But there is a more fundamental problem. You say you want to analyze protein sequences. BERT is intended for normal text. Why do you think this will work in the first place?

[XT]

I just want a try and use the bert to extract features is easy. Usually, many articals use CNN,LSTM or other varieties to extract protein sequences data, the precision of sota model is just 0.69. When I use features extracted by bert, the precision can up to 0.74, but the recall becomes lower.

[Rik]

Well, if it works it works, that's up to you. I already gave you what you need to know: a fully connected layer is equivalent to doing a matrix multiplication. You can even do the training fully inside Matlab with the mldivide, \ function.

[David Willingham]

Hi Rik,

I spoke with development on this, they state:

==

Reducing the features of BERT might not help and wouldn’t recommend doing that in that situation.

It seems that they are trying to extract features of protein sequence from BERT, which is something BERT is not trained for.

==

I note that you have tried Bert already. Are you able to provide some example code? This may help us provide more guidence on this.

[XT]

I have uploaded the code and data. I dont't know whether the loss function is correct

https://drive.matlab.com/sharing/8cd15796-cc2b-43fd-83a5-bbd7205a661c