Neural Network is not reducing loss during trainning

Hello, I am working in a project that describes how to estimate SoC using Neural networks. The example runs fine with provided dataset and the output Network does a really great job on predicting SoC. https://www.mathworks.com/help/deeplearning/ug/predict-soc-using-deep-learning.html

However, what I am doing is utilizing external data to train the Net and then observe what are the results after trainning with different data.

The problem is that I am not sure if my data need to be normalized, because the loss value is not reducing during trainning. And because of that, the model won't be able to predict the output accurately.

Attached is the example dataset, first 3 columns are feature inputs and last column is the output

Thanks in advance for your help!

[Taylor]

Normalizing data is not always a requirement when working with neural networks, but it is generally beneficial. For example, the tanh activation function you are using is sensitive to scale of the input data, so normalizing the inputs will ensure the data is in a reasonable range for the activation function to operate as intended. Also, while your first two features are both roughly normal distributions, they have means that differ by a few orders of magnitidue. Because of this, one feature may dominate the response of the network, effectively rendering the others useless. So in short, I would recommend normalizing your data and attempting to retrain the network.

[Axel]

Thank you Taylor, I appreciate your help.

When normilizing the data is there any rule of thumb that I can use to make sure the data is properly normilized ?

When you say that tahh is sesnitive to the scale, that means that it's best to have attirbutes in a range of values maybe just from 0 - 5 as an example right ?

Also, can we have negative values for input parameters?

For output parameter, do I also need to scale let's say to have an output in between 0 - 1. (I would need to scale values as well.).

I am following this link:

https://medium.com/@spinjosovsky/normalize-data-before-or-after-split-of-training-and-testing-data-7b8005f81e26#:~:text=When%20working%20with%20machine%20learning,dataset%20to%20a%20standard%20range.

[Taylor]

Generally, you want to normalize your data based on statistics from the training set only to avoid overfitting/data leakage (as described in the link you sent).

It is also useful to normalize to zero mean and unit variance to bring all features to a similar scale (assuming you want each feature to contribute equally). Ultimately it depends on what you want out of your layer in terms of how often is activates the next layer, but let's consider the outputs of tanh for values 0-5. Often, the threshold for a tanh activation function is "0". Every value in the range 0-5 would lead to activation of the next layer.

There isn't necessarily an issue with having negtive values fed into your layers. The ReLU activation function will reduce all negative inputs to zero, but other activation functions like tanh and leaky ReLU will preserve some signedness.

There are plently of details beyond this you can get caught up in, but, at the end of the day, my recommendation is to normalize your data. In the example your following, they normalize the data to be zero-centered. That's a great place to start. From there you can try other normalization methods if you'd like.

Answer 1

[Ganesh]

Hi @Axel,

I understand that you are unable to train your Neural Network model as your model loss does not reduce with training epochs. The following are possible solutions to your problem, which might be able to resolve the issue.

1. Normalization: Kindly normalize your data before feeding it to the Neural Network Model. To answer your question, the best normalization would be to keep the values bound between 0 and 1. Normalize the entire dataset, features and labels. In your case, the normalization can happen after splitting test set, as in a real world scenario, SOC decreases, and you will be unable to obtain global minimum value, while testing the network. Please refer to the documentation for understanding how to Normalize data on MATLAB: https://in.mathworks.com/help/matlab/ref/double.normalize.html
2. Learning Rate: Sometimes a very high or a very low learning rate can affect the model and not allow the model to learn correctly. I suggest you to modify your learning rate and then training your model to validate it's effect on your model.
3. Activation Function: Ensure that you have the right activation functions in place. e.g. do not use "softmaxlayer" as your last activation function as it is used in Classification Problems only. Do play with Activation Functions so that incompatible Activation functions arent being used. Please refer to the docuentation for understanding different activation functions usable in matlab: https://in.mathworks.com/help/deeplearning/ug/list-of-deep-learning-layers.html
4. Try to overfit the data, this is not suggested in practice, but can help you in debugging the issue, as you can understand if the problem lies in the data or in the code.

Hope this helps!