How to select model after ANN modelling in MATLAB?

I am doing ANN regression modelling using nntool in MATLAB. To evaluate my model after training, I calculated the mean absolute percentage error (MAPE) of the train, test and validation set and also the overall mean mape of the sets.

Using one hidden layer with 8 neurons, and softmax and purelin activation functions in hidden and output layer, I found the following MAPE results:

• Train MAPE: 20.41057169%,
• Test MAPE: 29.81276326%,
• Val MAPE: 34.91220059%.

In another architecture with one hidden layer with 100 neurons, and logsig and tansig activation functions in hidden and output layer, I found the following MAPE results:

• Train MAPE: 0.349799425%,
• Test MAPE: 35.47232397%,
• Val MAPE: 34.87608854%.

My question which model can be considered as a generalized model? Which one should I use? Do I need to change the architecture and try more?

Any suggestions are welcome. Thanks for reading.

Answer 1

[Shreeya]

Hello @Mohammad Mainul

According to my understanding, the second architecture seems to be overfitting the training data, as suggested by the difference between the training and testing MAPE values. The first architecture can be considered as a better generalized model. But hyperparameter tuning can help reduce MAPE while avoiding overfitting. Refer to the documentation below for hyderparameter optimization.

https://www.mathworks.com/help/stats/hyperparameter-optimization-in-classification-learner-app.html#mw_7a976371-1a5c-4c27-a258-563c9883a255

Bayesian optimization can be a good starting point. You can refer to the documentation below for more details:

https://in.mathworks.com/help/stats/bayesian-optimization-workflow.html

Hope this helps.