What is a good validation RMSE value?

Hi everyone

I am new to MATLAB and modelling. I have a LSTM deep learning model that I am experimenting with. I need some help in determining if the model is a good fit for my data. I am not sure where to start so I will start with validation RMSE. If there are other metrics to look at please let me know.

I have three versions of the model. There are close to 3000 data points used in the model. 80% is used for training. 20% is used for testing/validation.

Model 1 22 sec, validation RMSE = 0.35918

Model 2 34 sec, validation RMSE = 0.065824

Model 3 41 sec, validation RMSE = 0.50482

My elementary knowledge of modelling tells me that the lower the RMSE the better. So, if I use this approach then Model 2 is the winner. Is this enough? Should I be looking at other ways to assess quality of fit?

Here is some basic information about the data set:

Average 17.45502778

Min 8.8901784

Max 74.563979

Standard Deviation (Sample) 6.500568667

Standard Deviation (Population) 6.499343878

Thank you

Answer 1

[Jacob Mathew]

Hi Manny,

Assuming you have calculated the Root Mean Square Error on the original data itself, you can consider calculating the Normalised RMSE which is denoted using the formula:

where,

• n is the number of observations.
• are the observed values.
• are the predicted values.
• and are the maximum and minimum observed values, respectively

This effectively calculates the RMSE accounting for the variability in your data. In your case, this turns out to be:

• 0.005469 for Model 1
• 0.001002 for Model 2
• 0.007686 for Model 3

All 3 values are close to zero which imply the predictions have little error against the observered value. This generally hints that the model is well fitted to the data provided the data is sampled randomly to create the test, train and validation data and appropriate data preprocessing was done.

[Manny]

Thank you @Jacob Mathew. This is very helpful.

You made a comment re how the RMSE was calculated. I am not sure either myself. MATLAB produced a screen that shows the results. Here it is:

Does this address your comment?